Fact-checking AI's confident answers

The confidence problem

Language models are trained to produce fluent, confident text. That training doesn't distinguish between things the model knows well and things it's making up. A hallucinated statistic and a real one read identically — same sentence structure, same declarative tone, same apparent certainty.

This is not a bug that will be fixed in the next update. It's inherent to how the models work. The only defence is a human review habit that targets the right things.

What to verify (and what to skip)

You don't need to verify everything — you need to verify the things that would matter if wrong. Apply different levels of scrutiny based on consequence:

• Verify every time: specific numbers and statistics, named people and their roles, citations and quotes, legal or regulatory claims, medical facts, pricing and availability, dates and timelines.
• Spot-check: historical claims, scientific consensus, named processes or frameworks (do they exist as described?).
• Trust with minimal checking: general explanations of well-established concepts, structural suggestions (outlines, frameworks), tone and style choices, brainstormed ideas.

The three-step verify routine

For any claim in the "verify every time" category:

• Step 1 — Ask the model itself. "Are you confident this is accurate? Is there anything here you're uncertain about?" This sometimes surfaces doubt the model didn't volunteer. It also sometimes doesn't — but it takes five seconds and occasionally saves you.
• Step 2 — Search independently. Put the specific claim into a search engine (or a more recent AI tool with web access). You're looking for corroboration from a source you trust — a news outlet, government website, academic source, or the organisation directly. One good source is enough for most purposes.
• Step 3 — Check the date. Even a correct fact can be outdated. If the claim involves anything that changes — rates, laws, prices, statistics — confirm when the source was published and whether it reflects the current situation.

A useful trick: ask for uncertainty

Build uncertainty disclosure into your prompts by adding: "Flag any claims you're not fully confident about, and note where I should verify independently." Some models honour this consistently; others don't. But when it works, it pre-sorts which parts of the output deserve the most scrutiny.

Another useful prompt: after getting an answer, ask "What's a counterargument or alternative view to what you just said?" This forces the model to surface evidence against its own output — useful for anything where you want a balanced view.