Treat an AI answer as a draft to check, not as finished fact. The check has a name you will remember, Claim → Source → Confidence: find the claims, verify the risky ones against a real source, and mark what is safe to go. By the end you will have caught the invented bits while still on your screen.
To catch made-up facts in an AI answer, list every checkable claim (names, numbers, dates, quotes, links), sort them by risk, open a primary source yourself for the risky ones rather than asking the same chat, then label each claim use, soften or cut before you send.
Who this is for
- People past their first chat with Claude or ChatGPT, who already use AI for research, emails, posts, client notes or planning.
- Anyone who has pasted an AI answer somewhere and later found that a name, date, number or source was invented.
- Not for absolute beginners still learning how to open the app. If that is you, start with your first hour with Claude and come back.
What you will do
In about two minutes you will run a repeatable check on one real AI answer: spot the claims, stress-test the risky ones, and mark what you are willing to send. You will finish with a paste-ready mini prompt and a small habit you can keep.
Why AI still makes things up
The tool is not lying to you. It was built to continue helpful text and answer rather than hesitate, and its own makers have shown why: models are rewarded, in training and in testing, for guessing over admitting they are unsure, so a confident guess scores better than an honest "I do not know" (the research behind it).
That means a confident tone is free and unearned. Citations, URLs, quotes and "as of 2024" lines can all be produced the same way, true-sounding whether or not they are true. So the skill is a small one: know which few bits need a check before they leave your screen, and let the rest go.
The two-minute check
Three moves, in order: Claim, then Source, then Confidence. Find the claims, check the risky ones against a real source, label what is safe to send. Do it live on one answer now, not on a demo.
Step 1. Pick one answer you will actually use
- Open Claude or ChatGPT. Whichever you already use.
- Ask it something you would really send or publish this week. A short brief, a post with a stat in it, a quick research reply, a product comparison.
- Keep the reply on screen. You are checking this reply, the one you would actually use, not a practice one.
Step 2. Underline the claims
Go through the answer and mark every statement that could be wrong if it were invented:
- Names of people, companies, tools or laws.
- Numbers, prices, dates, percentages, and any "studies say".
- Quotes, links, paper titles, and "according to".
- How-to steps that depend on a real menu, screen or policy.
Leave the pure taste alone. "This draft feels warmer" is an opinion, not a fact claim, so it is not something to check here.
Step 3. Sort the claims by risk
- High. Anything that could embarrass you, mislead a client, or land you in trouble if it is wrong: legal, financial, medical, or a public number with your name on it.
- Medium. Public facts you could settle in one search: who founded a company, when a tool launched.
- Low. Soft framing with nothing checkable in it: "many teams struggle with this".
Only the high and medium claims go through the full loop today. This is the step that keeps the whole thing to two minutes, so do not skip it and try to check everything.
Step 4. Source pass: one check per risky claim
This is the one move professionals do differently from everyone else. When a fact-checker judges a page, they do not sit and study the page itself. They leave it and check the claim somewhere else, a habit researchers call lateral reading.
In a Stanford study the checkers who did this ran rings around historians and students who stayed put and were fooled by official-looking sites (the study). An AI answer is just another unfamiliar page. Do not ask the same chat to prove itself. For each high or medium claim:
- Ask where it would be true if it were true. Official docs, the primary site, the actual legislation, your own files.
- Open that source yourself. In a new tab, not by asking the chat that wrote the claim.
- Mark it Confirmed, Wrong or Unclear. Move on. You are checking, so keep it quick.
Click the link. If the answer gave a URL or a paper title, open it. Made-up citations are common enough that a court has fined lawyers for filing a brief built on cases an AI had invented, cases that did not exist (Mata v. Avianca).
Faster still, paste the exact title, quote or citation into a search engine inside quotation marks. If nothing sensible comes back, it is very likely invented, and a link that will not open or a paper you cannot find is the cheapest catch there is: treat the claim as unconfirmed.
Step 5. Label your confidence before you hit send
Go back through the answer and annotate it with three plain labels you keep for yourself:
- Use. Confirmed against a source you opened.
- Soften. Likely true but you did not verify it. Rephrase as "reportedly" or "worth checking", or cut the exact number.
- Cut. Wrong or unclear. Delete it before you send.
The sharpest version of this is to hand the answer to a fresh set of eyes. Open a new chat (or, in Claude Code, spawn a subagent), paste in only the answer, and run the prompt below.
A blank-slate chat cannot inherit the confident blind spots that produced the mistake, so it beats asking the same chat to mark its own homework. Treat what it marks Solid as a starting point for your own check, and run Claim, Source, Confidence yourself on anything it marks Check first, because a model cannot fully vouch for another model's output.
Review the answer below with completely fresh eyes and stress-test it for made-up facts, before I use any of it. Do not assume anything in it is correct. Only audit it, do not rewrite it or answer it again. Assume at least one claim in it is wrong, and go and find it. Work in this order: 1. Pull out every checkable claim: names, numbers, dates, quotes, links, prices, product features, and anything like "studies show" or "according to". Quote each claim exactly as you wrote it, do not paraphrase it into something safer. 2. For each claim, give me a row with these columns: - Claim (your exact words) - Type (name, number, date, quote, link, or feature) - Risk (High if it could mislead me or embarrass me, Medium if it is a public fact I could settle in one search, Low if it is soft framing with nothing checkable) - Verdict: Solid (you can point me to a real primary source that would confirm it) or Check first (you cannot, or you are not certain it is real, including anything you may have filled in yourself) - How to check it (the exact thing I should search, or the primary source I should open, in a few words) 3. Give links, citations and quotes the hardest look, since a made-up source is the cheapest lie to catch. If you cannot be sure a link or citation points to a real source, mark it Check first. 4. Sort the rows by risk, highest first. 5. End with a one-line verdict: the claims I must check before I send this, and the ones that are safe to use as they are. Rules: never invent or guess a source, a link or a citation. Never mark a claim Solid just because it sounds right. When in doubt, mark it Check first.
Step 6. Send it, with the habit on
Take the cleaned version and actually use it: send it, schedule it, or save it as a draft. Note what you cut. That is the win, a real thing that goes out with the made-up bits stripped.
Make it stick
- The default rule. If you would be upset to be wrong about it in public, run Claim, Source, Confidence first.
- Practise once a week. Deliberately ask something factual and run the triage, so the loop stays fast and you keep the muscle.
Where the line is
- AI can be useful and wrong in the same paragraph. The good bits and the invented bit sit side by side and read the same, which is exactly why you check the claims rather than the vibe.
- "Sounds right" is not a source. You are still the one responsible for whatever you publish or send, and the check moves that from hoping to knowing.
- This is not professional advice. For anything legal, medical or financial, a two-minute check is where you start, not where you finish. Take the real decision to a qualified human.
Common mistakes
- Only checking the tone. Catching the phrasing that sounds generated and waving through the numbers and names is checking the wrong layer. The facts are where the damage is.
- Asking the same model "are you sure?" The chat that wrote the draft is the worst referee of it. Models tend to cave or double down under pushback for reasons that have nothing to do with the truth, and they cannot reliably correct their own reasoning (the research). Check somewhere it cannot follow you.
- Verifying nothing because the answer was long and confident. Length and confidence are free. They are not evidence.
- Checking every soft sentence and burning twenty minutes. Triage first. Only the high and medium claims earn a check.
Questions people ask
How can I verify AI answers?
List every checkable claim in the answer (names, numbers, dates, quotes, links) and sort them by risk. Open a primary source yourself for the risky ones rather than asking the same chat, then mark each claim use, soften or cut before you send.
Isn't this just "don't trust AI"?
No. Trust the useful parts, after you have checked the risky claims. Treat the output as draft data, not finished fact, and the check tells you which bits need a look.
How is this different from spotting AI writing style?
Style tells catch phrasing that sounds generated, and there is a whole guide to those tells. This catch is about invented facts, the wrong dates, fake studies and made-up features, before you hit send.
Do I need a paid plan or special tools?
No. Your browser, your usual Claude or ChatGPT chat, and one search tab are enough for the two-minute check.
Do the two-minute check on one real answer now
Do not wait for a big research task. Open the last answer you were about to use, run Claim, Source, Confidence on it, and see what you cut. The first invented thing you catch before it went out is the moment this becomes a habit rather than a guide.