Saram ConsultingYou deploy a fact-checking LLM to catch hallucinations. It confidently flags three claims as false. Two of them were actually correct. The one real error? It missed it.
Sound familiar? You’ve just hit the most expensive trap in AI engineering: using one probability engine to verify another probability engine, with no connection to reality between them.
The core paradox
When LLM B “fact-checks” LLM A using only its parametric memory, it’s running the same token-prediction process. It’s not looking something up. It’s generating text that sounds like verification. Both models draw from overlapping web corpora, share similar biases, and optimize for the same objective: produce plausible-sounding output.
Three failure modes make this particularly dangerous:
Correlated errors. If Model A gets a date wrong because its training data was wrong, Model B likely learned the same wrong date. Consensus between models trained on similar data is not independent confirmation — it’s correlated noise.
Sycophancy. LLMs are fine-tuned to be helpful, which often translates to agreeing with confident-sounding input. If the original text reads authoritatively, the “fact-checker” hallucinates a justification to match that confidence. If you prompt it to “find hallucinations,” it invents false positives to satisfy you. Either way, you’re not getting truth — you’re getting compliance.
Same task, same failure. Generating a fact from memory and verifying a fact from memory are the same underlying computation. You haven’t introduced new information. You’ve just run the same flawed process twice and called the output “verification.”
The result: a single hallucination from Model A is bad. But when Model B “confirms” it, you now have “independent verification” of a false claim. That’s more dangerous than no verification at all, because it converts uncertainty into false confidence.
The fix: ground the checker in reality
The solution is architectural, not prompting. You must change the fact-checking LLM’s job from recalling truth to comparing claims against retrieved evidence.
Think of it this way: Model A is a lawyer writing a brief. The fact-checker should not be another lawyer opining from memory. It should be a paralegal whose only job is to go to the library, pull the specific books, read the exact paragraphs, and put them on your desk.
If the fact-checker cannot put a verifiable, citable source on your desk, you have learned nothing — except that both models are hallucinating.
The five architectural fixes
1. Atomic claim decomposition
LLMs cannot reliably evaluate multi-sentence paragraphs as single blocks. A paragraph like “Apollo 11 landed on July 20, 1969, carrying three astronauts, including Buzz Lightyear” contains three distinct factual claims. The checker gets confused and either accepts the whole thing or rejects it entirely based on one wrong detail.
Break every output into independently verifiable atomic claims before checking:
Original: "The EMA published their AI framework in January 2026,
covering all 27 member states with binding requirements."
Atomic claims:
1. The EMA published an AI framework.
2. The publication date was January 2026.
3. The framework covers all 27 EU member states.
4. The framework contains binding requirements.Each claim gets checked independently. This is the technique behind FActScore and SAFE benchmarks — and it’s the single biggest improvement you can make to any verification pipeline.
2. Anchor every check to retrieved evidence
Never ask: “Is this claim true?”
Always ask: “Does Source Document X explicitly support, contradict, or leave unmentioned this claim?”
This shifts the task from knowledge retrieval (high hallucination risk) to reading comprehension (low hallucination risk). The checker is no longer generating facts — it’s comparing a claim against text you provided.
Three grounding patterns, ranked by reliability:
| Pattern | How it works | Reliability |
|---|---|---|
| RAG pipeline | Retrieve relevant docs, feed to verifier with claim | High |
| Web search | Search-enabled model fetches sources, then reasons over them | High |
| Direct API/DB lookup | Query authoritative databases programmatically | Very high |
The critical constraint: the verifier can only select from retrieved sources. It cannot invent new citations. This “constrained attribution” prevents the checker from hallucinating its own supporting evidence.
3. Force exact quotes, not vibes
Never accept a bare verdict. If the checker says “SUPPORTED” or “CONTRADICTED,” require it to produce the exact verbatim quote from the source text that proves the point.
If it cannot extract a direct quote, it must label the claim as UNSUPPORTED — not “false,” not “probably wrong,” but “we cannot verify this from the provided evidence.”
This makes the verification falsifiable. You can check whether the quote exists in the source. You can check whether the quote actually says what the checker claims. Without this requirement, the checker is just generating another confident-sounding paragraph with no accountability.
4. Use structured classification, not binary true/false
Binary true/false is the wrong framing. The checker needs three categories at minimum:
| Category | What it means | What to do |
|---|---|---|
| VERIFIED | Direct quote from source supports this | Accept |
| CONTRADICTED | Source text explicitly states the opposite | Reject |
| UNSUPPORTED | Source does not mention this | Flag for human review |
The UNSUPPORTED category is critical. “I couldn’t find evidence” is not the same as “this is false.” And “the evidence doesn’t contradict it” is not the same as “the evidence proves it.” Without this escape hatch, the checker is forced into false precision — and false precision is where hallucination hides.
5. Gate the output with deterministic code
For things LLMs are worst at — math, dates, logic, exact quotes, citation verification — don’t use LLM judgment. Use code.
| Claim type | Verification method |
|---|---|
| Math / statistics | Python code interpreter |
| Paper citations | DOI lookup, URL resolution |
| Code correctness | Compiler / test suite |
| Regulatory text | Direct corpus query |
| Dates / numbers | API or database lookup |
Pass the checker’s structured output into code. If any claim flags as CONTRADICTED or UNSUPPORTED, reject the response programmatically rather than letting another LLM decide what to do with the verdict.
The verification architecture
Here’s the full pipeline, from generation to final verdict:
CANDIDATE OUTPUT
│
▼
┌──────────────────┐
│ Claim Extraction │
│ (atomic facts) │
└────────┬─────────┘
│
atomic claims
│
▼
┌──────────────────────┐
│ Evidence Orchestrator │
└──────────┬───────────┘
│
┌───────────────┼───────────────┐
▼ ▼ ▼
Vector RAG Web Search API / DB
│ │ │
└───────────────┼───────────────┘
│
┌────────────────┐
│ Evidence Bundle │
└───────┬────────┘
│
┌────────────┼────────────┐
▼ ▼ ▼
Checker A Checker B Checker C
(model 1) (model 2) (model 3)
│ │ │
└────────────┼────────────┘
│
▼
┌─────────────┐
│ Adjudicator │
└──────┬──────┘
│
▼
┌─────────────────────┐
│ Deterministic Gates │
└──────────┬──────────┘
│
▼
VERDICT The key design principle: separate retrieval from reasoning. Don’t make one LLM responsible for searching, interpreting, verifying, citing, and adjudicating. Each stage is a different concern with different failure modes.
Multi-model triangulation: what works and what doesn’t
Running the same claim through multiple models from different providers is better than one model — but with a critical caveat.
What works: Using disagreement as a signal. If Checker A says VERIFIED with a real source and quote, and Checker B says FALSE from memory with no source, A wins every time. Disagreement between a grounded checker and an ungrounded checker tells you exactly where to focus.
What doesn’t work: Treating agreement between ungrounded models as proof. If three models all trained on similar web data agree on a fact, that’s weak evidence. They may share the same training-data error. Agreement between models is not the same as agreement with reality.
The practical rule: Use models from different labs (different training data, different architectures). Set temperature to 0 for maximum determinism. And flip the adversarial framing — tell the checker to try to disprove the claim, not confirm it. A model looking for disproof that fails is far more trustworthy than a model looking for confirmation that succeeds.
The verification confidence ladder
Not all verification is equal. Calibrate your trust:
| Level | Method | Trust |
|---|---|---|
| 0 | Single LLM assertion from memory | Unreliable |
| 1 | Second LLM agrees from memory | Unreliable |
| 2 | Multiple LLMs agree from memory | Weak |
| 3 | LLM + retrieved evidence | Moderate |
| 4 | LLM + authoritative source with citation | Strong |
| 5 | Authoritative evidence + code verification | Very strong |
| 6 | Human review + evidence + audit trail | Definitive |
For regulated environments, map risk to required level:
- Low-risk informational claim → Level 3-4
- Medium-risk recommendation → Level 4-5 + human spot-check
- High-stakes decision (GxP, safety, regulatory) → Level 5-6 mandatory
When the checkers disagree
This is the most important scenario to get right, because it’s where most people make the worst mistake: asking a fifth LLM to break the tie.
Don’t. When models conflict:
- If one has a real source and the other doesn’t — the sourced one wins.
- If both have sources but disagree — open the sources yourself. One is often misreading or quoting an outdated blog.
- If neither can find a source — it’s not a hallucination. It’s unverifiable. Mark it accordingly and don’t use it for anything important.
Absence of evidence is not evidence of hallucination. Most false “hallucination!” flags happen because the checker had bad search, not because the original was wrong.
Red flags your checker is hallucinating
Watch for these patterns in the verification output:
- Vague corrections: “Some experts disagree” without naming them or citing a source
- Invented citations: URLs that 404, paper titles that don’t exist, publishers that don’t match
- Micro-shifts: Changing “March 14, 2022” to “early 2022” — sounding precise while erasing accuracy
- Overconfident negation: “This is false” without explaining what the truth actually is
- Perfect agreement: 100% agreement on a long, complex text means the checker is sycophantic, not rigorous
The bottom line
You cannot eliminate hallucination by adding more LLMs to the stack. An LLM cannot fact-check another LLM into truth any more than one echo can correct another echo.
The architecture that actually works breaks the closed loop:
Generate → Decompose → Retrieve evidence → Compare → Gate programmatically → Human review for flagged items
The mental model shift: stop thinking “how do I make an LLM fact-check another LLM?” Start thinking “how do I build an evidence verification system in which LLMs are replaceable reasoning components?”
LLMs should sit below evidence and controls in your architecture — not above them. Use them to reason about reality, not to define reality. That one principle, applied consistently, is the difference between a hallucination amplifier and a hallucination detector.
Research notes: [[LLM-as-Judge-Fact-Checking-Architecture-2026]]