Treat an AI answer as a draft full of data to check, not as finished fact. The check has a name you will remember, Claim → Source → Confidence: find the claims, verify the risky ones against a real source, and mark what is safe to go. By the end you will have caught the invented bits while they were still on your screen.
To catch made-up facts in an AI answer, list every checkable claim (names, numbers, dates, quotes, links), sort them by risk, open a primary source yourself for the risky ones rather than asking the same chat, then label each claim use, soften or cut before you send.
Who this is for
- People past their first chat with Claude or ChatGPT, who already use AI for research, emails, posts, client notes or planning.
- Anyone who has pasted an AI answer somewhere and later found that a name, date, number or source was invented.
- Not for absolute beginners still learning how to open the app. If that is you, start with your first hour with Claude and come back.
What you will do
In about two minutes you will run a repeatable check on one real AI answer: spot the claims, stress-test the risky ones, and mark what you are willing to send. You will finish with a paste-ready mini prompt and a small habit you can keep.
Why AI still makes things up
The tool is not lying to you. It was built to continue helpful text and to answer rather than to hesitate, and its own makers have shown why: models are rewarded, in training and in testing, for guessing over admitting they are unsure, so a confident guess scores better than an honest "I do not know" (the research behind it). That means a confident tone is free and unearned. Citations, URLs, quotes and "as of 2024" lines can all be produced the same way, true-sounding whether or not they are true. So the skill is a small one: know which few bits need a check before they leave your screen, and let the rest go.
The two-minute check
Three moves, in order: Claim, then Source, then Confidence. Find the claims, check the risky ones against a real source, label what is safe to send. Do it live on one answer now, not on a demo.
Step 1. Pick one answer you will actually use
- Open Claude or ChatGPT. Whichever you already use.
- Ask it something you would really send or publish this week. A short brief, a post with a stat in it, a quick research reply, a product comparison.
- Keep the reply on screen. You are checking this reply, the one you would actually use, not a practice one.
Step 2. Underline the claims
Go through the answer and mark every statement that could be wrong if it were invented:
- Names of people, companies, tools or laws.
- Numbers, prices, dates, percentages, and any "studies say".
- Quotes, links, paper titles, and "according to".
- How-to steps that depend on a real menu, screen or policy.
Leave the pure taste alone. "This draft feels warmer" is an opinion, not a fact claim, so it is not something to check here.
Step 3. Sort the claims by risk
- High. Anything that could embarrass you, mislead a client, or land you in trouble if it is wrong: legal, financial, medical, or a public number with your name on it.
- Medium. Public facts you could settle in one search: who founded a company, when a tool launched.
- Low. Soft framing with nothing checkable in it: "many teams struggle with this".
Only the high and medium claims go through the full loop today. This is the step that keeps the whole thing to two minutes, so do not skip it and try to check everything.
Step 4. Source pass: one check per risky claim
This is the one move professionals do differently from everyone else. When a fact-checker judges a page, they do not sit and study the page itself. They leave it and check the claim somewhere else, a habit researchers call lateral reading, and in a Stanford study the checkers who did this ran rings around historians and students who stayed put and were fooled by official-looking sites (the study). An AI answer is just another unfamiliar page. Do not ask the same chat to prove itself. For each high or medium claim:
- Ask where it would be true if it were true. Official docs, the primary site, the actual legislation, your own files.
- Open that source yourself. In a new tab, not by asking the chat that wrote the claim.
- Mark it Confirmed, Wrong or Unclear. Move on. You are checking, so keep it quick.
Click the link. If the answer gave a URL or a paper title, open it. Made-up citations are common enough that a court has fined lawyers for filing a brief built on cases an AI had invented, cases that did not exist (Mata v. Avianca). Faster still, paste the exact title, quote or citation into a search engine inside quotation marks. If nothing sensible comes back, it is very likely invented, and a link that will not open or a paper you cannot find is the cheapest catch there is: treat the claim as unconfirmed.
Step 5. Label your confidence before you hit send
Go back through the answer and annotate it with three plain labels you keep for yourself:
- Use. Confirmed against a source you opened.
- Soften. Likely true but you did not verify it. Rephrase as "reportedly" or "worth checking", or cut the exact number.
- Cut. Wrong or unclear. Delete it before you send.
To make the claims easier to spot in the first place, you can have the model flag its own before you start. Paste this above or after its answer. Treat what it marks Confirmed as a starting point for your own check, not the check itself, because a model cannot reliably vouch for its own output.
Before I use your answer, list every checkable claim you made: names, numbers, dates, quotes, links, product features. For each one, mark it Confirmed only if I could verify it from a primary source, otherwise mark it Unverified. Do not invent sources or links. If you are not sure, mark it Unsure and do not guess.
Step 6. Send it, with the habit on
Take the cleaned version and actually use it: send it, schedule it, or save it as a draft. Note what you cut. That is the win, a real thing that goes out with the made-up bits stripped.
Make it stick
- The default rule. If you would be upset to be wrong about it in public, run Claim, Source, Confidence first.
- Practise once a week. Deliberately ask something factual and run the triage, so the loop stays fast and you keep the muscle.
Where the line is
- AI can be useful and wrong in the same paragraph. The good bits and the invented bit sit side by side and read the same, which is exactly why you check the claims rather than the vibe.
- "Sounds right" is not a source. You are still the one responsible for whatever you publish or send, and the check moves that from hoping to knowing.
- This is not professional advice. For anything legal, medical or financial, a two-minute check is where you start, not where you finish. Take the real decision to a qualified human.
Common mistakes
- Only checking the tone. Catching the phrasing that sounds generated and waving through the numbers and names is checking the wrong layer. The facts are where the damage is.
- Asking the same model "are you sure?" The chat that wrote the draft is the worst referee of it. Models tend to cave or double down under pushback for reasons that have nothing to do with the truth, and they cannot reliably correct their own reasoning (the research). Check somewhere it cannot follow you.
- Verifying nothing because the answer was long and confident. Length and confidence are free. They are not evidence.
- Checking every soft sentence and burning twenty minutes. Triage first. Only the high and medium claims earn a check.
A couple of quick questions
Isn't this just "don't trust AI"?
No. Trust the useful parts, after you have checked the risky claims. Treat the output as draft data, not finished fact, and the check tells you which bits need a look.
How is this different from spotting AI writing style?
Style tells catch phrasing that sounds generated, and there is a whole guide to those tells. This catch is about invented facts, the wrong dates, fake studies and made-up features, before you hit send.
Do I need a paid plan or special tools?
No. Your browser, your usual Claude or ChatGPT chat, and one search tab are enough for the two-minute check.
Do the two-minute check on one real answer now
Do not wait for a big research task. Open the last answer you were about to use, run Claim, Source, Confidence on it, and see what you cut. The first invented thing you catch before it went out is the moment this becomes a habit rather than a guide.