PPassTheCertExam

Study guides / CCAO-F / Domain 2

Output Evaluation and Validation · Lesson 2 of 7

2.2 — Fact-Checking and Citation-Traceable Answers

Cross-check claims across independent sources and verify quoted passages against their originals — key evaluation techniques for catching unreliable output before it's trusted.

Fact-checking a Claude-generated answer is a distinct skill from writing a good prompt for one. Even a well-grounded response deserves scrutiny before it's trusted for anything that matters, and two techniques do most of the work: cross-checking a claim against multiple independent sources before treating it as settled, and verifying that a quoted passage actually appears, in context, in the source it's attributed to rather than trusting the quotation marks on their own.

Cross-Checking Across Sources, Not Just One

A claim that's plausible-sounding but wrong is often plausible precisely because it resembles something true. Checking it against a single source doesn't protect you if that source is itself the origin of the error, or if you happen to pick the one source that agrees by coincidence. The more reliable approach is to check a claim against at least two independent sources and pay particular attention to disagreement: if two sources conflict, that disagreement is itself the finding — the honest answer is "sources disagree on this," not a confident pick of whichever one you saw first.

This matters most for claims that are surprising, unusually precise, or carry real consequences if wrong — a headline statistic that will be quoted publicly, a legal or regulatory claim, a number that will drive a budget decision. Routine, low-stakes claims don't need the same scrutiny as claims that will travel.

Verifying a Quote Means Opening the Source

Asking Claude to quote the exact passage it's drawing from, rather than paraphrasing, is a genuinely useful evaluation technique — it gives a reviewer something concrete to check instead of a vague summary. But it's a starting point, not a finish line. Quotation marks around a passage make it look traceable; they don't make it verified. A model can still generate a quote that reads exactly like something a source would say without that exact wording appearing in the source at all. Treating "it was in quotes" as equivalent to "it was checked" is the trap. The actual verification step is opening the cited source and confirming the quoted text appears there, in the context the answer implies — not just that similar-sounding words exist somewhere in it.

When you're reviewing many claims and don't have time to verify all of them with equal rigor, prioritize: verify the surprising ones, the precise ones, and the ones that will be repeated or acted on, before spending equal time confirming something mundane and low-stakes.

Key Concept

Two concrete evaluation techniques: cross-check claims against multiple independent sources and treat disagreement between them as a real finding, and verify every quoted passage by opening its cited source — quotation marks indicate traceability was attempted, not that it was completed.

Common Exam Distractor

Watch for an answer that treats "the response includes a quote" or "the response cites a source" as sufficient evidence of accuracy on its own. A citation or quotation is only as trustworthy as the verification step that follows it — an unverified quote is not meaningfully more reliable than an unverified paraphrase.

Exam traps

Practice question

Claude produces a research memo with several claims, each followed by a quoted passage attributed to one of three uploaded source documents. Before the memo is shared externally, what should the reviewer do?

  • A Independently open each cited source and confirm the quoted text actually appears there in context, prioritizing the most surprising or consequential claims Correct

    This is the concrete verification step — quotation alone shows traceability was attempted, not confirmed, and prioritizing by stakes makes limited review time count.

  • B Accept the memo as reliable since every claim already includes a direct quotation

    A quoted passage can be fabricated or subtly misattributed — quotation marks indicate an attempt at traceability, not completed verification.

  • C Check only the single claim that seems least surprising, since it's least likely to have been altered

    Verification effort is better spent on surprising, precise, or high-stakes claims, which are more valuable and more likely to matter if wrong — not on the least consequential one.

  • D Ask Claude to reformat the quotes in bold so they stand out more clearly in the memo

    Formatting changes don't verify anything — this addresses presentation, not whether the quoted content is actually accurate and traceable.

Build exercise: Verify Claude's Quotes Against Their Actual Source

Beginner · 20 minutes

You'll practice:

  1. In a Claude Project, upload two or three short reference documents (or paste them into project knowledge) containing a few distinct factual statements each. Ask a question whose answer draws on more than one document, and instruct Claude to quote the exact passage it's using for each claim before answering.

    This produces a response with concrete, checkable quotes rather than a paraphrased summary — the raw material this exercise verifies.

    You should see: A response with one or more quoted passages, each attributed to a specific source, followed by an answer built on those quotes.

    Hints 
    1. Make sure your documents actually disagree or add nuance to each other on at least one point, so cross-checking has something real to find.
    2. Ask a question that plausibly draws from more than one document, not just the first one.
    3. Note which claim in the response feels most surprising or consequential — that's the one to prioritize checking first.

  2. Open your source documents directly and manually confirm each quoted passage actually appears, word for word, in the context the response implies.

    This is the actual verification step — confirming a quote is real, not just quote-shaped — and it's the step that's easy to skip under time pressure.

    You should see: Either full confirmation that each quote matches its source exactly, or a case where the wording is close but not identical, or the context is subtly different from what the answer implies.

    Hints 
    1. Search for the quoted text directly in your source document rather than trusting your memory of it.
    2. Check not just the words but the surrounding context — a technically-accurate quote can still be used misleadingly if the surrounding meaning was different.
    3. If everything matches, that's a useful result too — write down what specifically made it easy to verify.

  3. Ask a follow-up question whose answer is not actually contained in your uploaded documents, and see whether Claude quotes something anyway or correctly states the information isn't in the provided sources.

    This tests the edge case that matters most for traceability: whether the system stays honest about what it doesn't have grounding for, rather than fabricating a plausible-sounding quote to fill the gap.

    You should see: Ideally, a response that clearly states the answer isn't found in the provided documents, rather than a fabricated quote presented as if it were sourced.

    Hints 
    1. Pick a question that's topically related to your documents but genuinely not answered by them, so the test is fair.
    2. If Claude does produce a quote, check extremely carefully whether it actually exists in your source text.
    3. Write one sentence on what this tells you about how much you can trust a quote-based answer without checking it.

Sources