Output Evaluation and Validation · Lesson 6 of 7

2.6 — Checking Semantic Consistency Across Paraphrases

Ask the same underlying question in different words and check whether the answers agree in substance — inconsistency here is a reliability red flag worth investigating.

A useful, low-effort reliability check doesn't require touching a second tool or writing any code: ask the same underlying question two or three different ways — different wording, same actual question — and read the answers side by side. If Claude's substantive answer changes depending only on how the question was phrased, that's a signal worth taking seriously, even when each individual answer sounds perfectly confident on its own.

Wording Differences vs. Substantive Differences

The check is only useful if you're comparing the right thing. Two answers can use completely different sentences, structure, and level of detail while agreeing in substance — that's a good, consistent result, and it's exactly what you'd expect from a reliable underlying answer expressed two ways. What actually matters is whether the substance changes: a different final number, a different yes-or-no conclusion, a different named recommendation, a caveat present in one answer and silently missing from another. A genuinely inconsistent pair of answers on the same underlying question is a real finding, not a stylistic quirk.

How to Run This Check Without Any Tooling

The most direct version: ask your real question, then ask it again in a separate message or a fresh conversation with different phrasing — a different sentence structure, a different framing, maybe from a slightly different angle ("is X legal in this state" vs. "what are the legal restrictions on X in this state"). Read both answers and specifically check whether the bottom-line conclusion matches.

A second, faster technique: paste both answers back into a single conversation and ask Claude to compare them and identify any substantive difference between the two. This is a narrower, more constrained task than asking a model to grade its own open-ended output (the self-grading risk from lesson 2.3) — comparing two fixed texts for concrete differences is more checkable than an open-ended quality judgment, since you can verify the comparison yourself by rereading both texts. Still, treat this as a helpful first pass, not a replacement for you actually reading both answers yourself, especially for anything consequential.

One caution when writing your paraphrases: avoid accidentally making one version a leading question that nudges toward a particular answer while the other stays neutral. If your two phrasings aren't actually asking the same thing with equal neutrality, a difference in the answers might reflect that instead of genuine unreliability.

Key Concept

Ask the same underlying question in two or three genuinely different phrasings and compare the substance of the answers, not the wording. Agreement in substance despite different wording is a good sign; a different bottom-line conclusion, number, or recommendation across phrasings is a reliability red flag worth independently verifying.

Common Exam Distractor

Watch for an answer that judges consistency by comparing word overlap or phrasing similarity between two answers. Two answers can be worded almost identically and still subtly disagree in substance, and two answers can be worded completely differently while agreeing perfectly — wording similarity is not the same thing as substantive agreement.

Exam traps

Judging consistency by how similar the wording of two answers looks, rather than whether the substance agrees

Wording similarity and substantive agreement are different things — two differently-worded answers can agree completely, and two similarly-worded answers can subtly disagree on the actual conclusion.
Treating one of two inconsistent answers as automatically correct because it's more detailed or came second

Neither detail level nor order indicates which answer is accurate — an inconsistency between paraphrased answers is a signal to verify independently, not a tiebreaker to resolve by guessing.
Writing paraphrased versions of a question where one phrasing subtly leads toward a particular answer

If the two phrasings aren't asking the same thing with equal neutrality, a difference in the answers may reflect the leading phrasing rather than genuine underlying inconsistency — the check only works if the paraphrases are fair.

Practice question

You ask Claude 'Is it legal for our company to record customer support calls without notifying the caller?' and get a confident 'No, this requires notification in most cases.' Later, you ask 'What are the legal requirements around recording customer calls?' phrased more generally, and the answer describes notification as depending heavily on which state or country the call takes place in, without a clear overall yes or no. What should you conclude?

A The two answers disagree on substance (a flat 'no' vs. a jurisdiction-dependent answer), which is a reliability red flag — verify the actual legal requirement independently before relying on either answer Correct
This is a genuine substantive inconsistency, not a wording difference — one answer gives a flat rule and the other says it depends, which is exactly the kind of disagreement this check is designed to surface and that warrants independent verification.
B Since the second answer is more detailed and mentions jurisdiction, it should be treated as the correct one
More detail doesn't establish which answer is accurate — treating the more elaborate answer as automatically correct is a guess, not verification.
C The two answers are consistent, since both are about call recording law
Sharing a general topic isn't the same as agreeing in substance — one gives a flat answer and the other describes it as jurisdiction-dependent, a real difference in the bottom-line conclusion.
D Ask the same question a third time in the exact same wording as the first time to get a tiebreaker
Repeating the identical phrasing doesn't test genuine consistency across paraphrases and doesn't resolve which underlying answer is actually correct — independent verification is needed instead.

Build exercise: Run a Paraphrase Consistency Check

Beginner · 20 minutes

You'll practice:

Ask the same underlying question multiple different ways
Distinguish a wording difference from a substantive one
Use Claude to help compare two answers for concrete differences

In claude.ai, ask a factual or explanatory question with some real nuance (not a single-word lookup) — for example, a question about a policy, a best practice, or a rule with exceptions. Then, in a separate message or fresh conversation, ask the same underlying question with meaningfully different wording and framing.

This produces the raw material for a consistency check — two genuinely differently-phrased answers to compare.

You should see: Two answers addressing the same underlying question, differing in wording and possibly in structure or emphasis.
Hints
1. ```
Choose a question where a nuanced or conditional answer is plausible, so there's real room for a substantive difference to show up.
```
2. ```
Change the sentence structure and angle of the second phrasing meaningfully, not just a synonym swap.
```
3. ```
Keep both full answers visible or saved so you can compare them side by side.
```
Read both answers yourself and write one or two sentences judging whether they agree in substance — same bottom-line conclusion, same key conditions or caveats — regardless of how differently worded they are.

This builds the actual judgment skill directly, rather than skipping straight to asking Claude to do the comparison for you.

You should see: A short written judgment noting either genuine agreement despite different wording, or a specific substantive difference (a different conclusion, a caveat present in one but not the other).
Hints
1. ```
Focus specifically on the bottom-line conclusion of each answer before comparing details.
```
2. ```
Note any caveat or exception that appears in one answer but not the other — that's a common place substantive differences hide.
```
3. ```
If the two answers look consistent, try a third phrasing that approaches the question from an even more different angle.
```
Paste both full answers into a new message and ask Claude to identify any substantive differences between them, not just wording differences. Compare Claude's comparison against your own judgment from step 2.

This tests using Claude for a narrower, more checkable comparison task, and lets you verify its comparison against your own reading rather than trusting it blindly.

You should see: A response identifying specific points of agreement or disagreement between the two answers, which you can then check against the answers themselves.
Hints
1. ```
Ask specifically for 'differences in substance, not wording' so the comparison doesn't get sidetracked by phrasing.
```
2. ```
Reread both original answers to confirm Claude's comparison is actually accurate, rather than accepting it at face value.
```
3. ```
Write one sentence on whether this comparison step would have been faster or more reliable than your own manual read from step 2.
```

Sources

Reused/adapted from Domain 4 fragment lesson 4-4-semantic-consistency, reframed as a no-code manual technique for business users (the original embedding/cosine-similarity approach requires code and doesn't fit this track)
Anthropic docs — Building Evals