Applying a “Sniff Test” or carrying out a “Sanity Check”, has always been an important part of any analysis or calculation. It should be your first check before diving deeper, and especially before sharing results with others.
I am not saying it was easy or foolproof, or even always possible; however, the onus was on you, since you came up with the answer and had often spent a lot of time arriving at it.
With the rapid rise of generative AI, you no longer need to do the heavy lifting; complex questions can receive intelligent-sounding answers in seconds, and it is tempting to accept them after a cursory glance. After all, we are busy, and the answers provided are intended to be coherent and phrased with a confident tone.
How To Apply a Sniff Test
I don’t think anyone would disagree with the need to ask ourselves, “does this answer make sense?“. Here are some suggestions:
- Does this align with what I already know?
- Is there a coherent thread to follow in the argument provided by the AI?
- If I am asked to explain the answer, can I do it in my own words?
- Can I find a trustworthy source that provides an alternative point of view or answer?
- If the answer was written by a brilliant, but recently graduated intern, would I have spent a similar amount of time reviewing it?
- Did the prompt I wrote steer the AI toward the answer I wanted?
- Did I refine the prompt through several iterations, or go with the first version because I liked the answer?
If the answer to any of these questions gives you pause, that’s your signal to dig deeper.
Risks of the Sniff Test
Like all techniques, the sniff has its own limitations:
- Bias, assumptions, or lack of domain knowledge can distort the Sniff Test.
- Correct but counterintuitive answers are often thrown out.
- It is easy to develop Sniff Fatigue after a period of questioning everything. Trusting AI out of convenience is a very easy slope to slip down.
- A Sniff Test is a first step, not a last line of defense, it doesn’t replace the need for a thorough review, that is appropriate for how you are going to use the answered provided.
Final Thoughts
Ask yourself Does this feel right? Does it make sense? Don’t take the easy way out.