Eval, Testing & Debugging · Lesson 2 of 2

8.2 - Building an Evaluation Harness

Set up a repeatable, fixed-dataset way to measure whether a prompt or model change actually improved things, instead of judging by a handful of manual spot checks.

Manual spot-checking - trying a prompt on a few examples and eyeballing the output - doesn't scale and doesn't catch regressions. It confirms the cases you happened to try worked; it says nothing about the cases you didn't try. A minimal evaluation harness replaces that gut check with three fixed components: a representative golden dataset (real or realistic inputs, including known-hard edge cases), a consistent grading method (exact-match where the task allows it, a rubric-based grade where it doesn't), and a repeatable runner that executes the same dataset through the same grading logic every time, so a score from one run means the same thing as a score from another. In effect, it's a regression test suite for a probabilistic system.

Building the Golden Dataset

Each example needs an input and either an expected answer (for exact-match tasks) or a grading rubric (for open-ended ones). The dataset should be weighted toward the inputs that actually matter in production - including the ones known to be hard. When a real failure shows up in production, the fix isn't just patching the prompt; it's also adding that exact failure as a new permanent case in the dataset, so a future change can never silently reintroduce it. Treat the dataset like code: version it, review changes to it, and never edit it in the same commit as the thing it's testing.

Choosing a Grading Method

Prefer exact-match or programmatic checks whenever the task allows it - a classification label, a specific extracted JSON field, a regex the output must satisfy, a numeric value within tolerance. These are cheap, deterministic, and unambiguous; there's no reason to reach for anything fuzzier when the task has one correct answer. For open-ended output - summary quality, tone, whether instructions were actually followed - exact match doesn't apply, so grading falls to either human raters or a model-graded rubric (LLM-as-judge).

LLM-as-Judge: Power and Pitfalls

The common pattern: a separate Claude call receives the original input, the output being graded, and a fixed rubric, then returns a score (pass/fail, or 1–5) plus a short justification. This scales far better than human grading and catches nuance that exact-match can't express. But it carries a real risk: a model grading outputs from its own model family - especially its own kind of output - can rate them more favorably than an independent judge would, a self-preference bias. The mitigations are structural, not aspirational: hold the rubric prompt fixed across comparisons, prefer a different or stronger model as the judge than the one under test, and periodically sample a batch of automated grades for a human to re-check against.

Common exam distractor

An answer that uses a model-graded rubric with no fixed rubric prompt and no periodic human check is a trap - 'model-graded' is not automatically trustworthy. The exam expects you to know that unchecked self-grading (particularly a model grading its own family's output) can inflate scores, and that a fixed rubric plus human spot-checks is the mitigation, not a reason to avoid LLM-as-judge altogether.

Keeping the Comparison Valid: One Variable at a Time

For a before/after comparison to mean anything, the test set and grading criteria must stay fixed while exactly one thing changes - the prompt, the model, or a single parameter. Changing the dataset at the same time as the prompt makes it impossible to attribute a score difference to either one specifically. Store every run's results with its full metadata - model ID, prompt version or hash, dataset version, timestamp - so any two runs can be diffed unambiguously later, and a dataset update is itself logged as a distinct, deliberate event rather than folded silently into an unrelated change.

Key concept

An eval set is only useful if it can catch a regression, not just confirm a win - deliberately include cases you're not confident will pass. A dataset that always scores 100% has stopped telling you anything.

Sample Size and Run-to-Run Variance

Claude's output is not perfectly deterministic even with identical inputs and default settings - a score can shift by a few points between two runs of the exact same prompt against the exact same dataset. A jump from 80% to 88% on a ten-example set could be a real improvement, or it could be noise from one or two examples flipping. Larger datasets average out individual-example volatility; running each version multiple times and comparing distributions (not single scores) is more reliable still. A single run on a small sample is fine for quick iteration during development, but it is not sufficient evidence to declare a regression fixed or a change a genuine improvement.

Running the Harness Repeatably

The runner itself is simple: iterate the fixed dataset, call the model/prompt version under test for each input, apply the grading method, aggregate the results into an overall score (and, ideally, a per-example pass/fail so you can see exactly which cases broke), and write the run to a versioned log. Every subsequent run diffs against a stored baseline instead of relying on someone's memory of how the last version performed - that's what turns evaluation from a one-off exercise into an actual regression gate a team can trust before shipping a prompt or model change.

Exam traps

Changing the evaluation test set at the same time as the prompt being tested, then comparing before/after scores.

If both the prompt and the test set changed, a score difference can't be attributed to either one specifically - a valid comparison needs the test set and grading criteria held fixed across the change being measured.
Building an eval set only from cases the current prompt is already confident about, with no deliberately hard or known-failure cases included.

A dataset with only easy, confirmatory examples will show a high pass rate regardless of real quality and can never detect a regression - it can only confirm what already works, not catch what's broken.
Using a model to grade its own (or its own family's) output as the sole check, with no fixed rubric and no periodic human spot-check.

This risks self-preference bias - a model can rate outputs from its own model family more favorably than an independent judge would. A fixed rubric prompt plus periodic human calibration checks are the standard mitigation, not skipping LLM-as-judge entirely.
Declaring a change an improvement (or a regression fixed) based on one run over a very small sample, without accounting for run-to-run variance.

Claude's output isn't perfectly deterministic - a few points of score movement on a small dataset can be noise rather than a real effect. Larger samples or multiple runs are needed before treating a score change as meaningful.

Practice question

A team rewrites a prompt and, to keep things simple, tests the new version against ten new example questions rather than the original 50-question set used to test the old version, reporting a higher score as proof of improvement. What's wrong with this comparison?

A Ten questions is too few to ever be statistically meaningful, regardless of anything else.
Sample size is a secondary concern here - the more fundamental problem is that the test set itself changed, which breaks the comparison entirely.
B The test set changed along with the prompt, so the score difference can't be attributed to the prompt change specifically. Correct
With two things changing at once - the prompt and the questions - there's no way to isolate which one caused the score difference, making the comparison invalid.
C Automated grading is never valid for prompt evaluation.
Automated grading, including model-graded rubrics, is a legitimate and common technique - the issue here is specifically the changed test set, not the grading method.
D Nothing is wrong - a higher score on any test set is valid evidence of improvement.
A score on a different, easier, or differently-composed test set isn't a valid before/after comparison at all.

Build exercise: Build a fixed eval harness with exact-match and LLM-as-judge grading, and run a valid before/after comparison

Intermediate · 50 minutes

You'll practice:

Build a small, fixed, representative test set with at least one deliberately hard case
Implement both exact-match and LLM-as-judge grading, and know when to use each
Run the same test set against two prompt versions and compare honestly, with results stored by metadata
Calibrate an automated judge against a human spot-check to catch self-preference bias

Write 8-10 test inputs for a task you have a prompt for, including at least one deliberately tricky edge case, plus an expected answer or grading rule for each. Save it as a JSON file that never changes once the comparison starts.

The edge case is what actually differentiates a useful eval set from a set that only confirms what already works, and freezing the file is what makes every later run comparable.

You should see: A fixed list of dicts, each with an input and either an expected_answer or a grading_rubric string, saved to disk (e.g. eval_set.json), that you will not edit again for the rest of this exercise.

Hints

What kind of task from your own work has a clear right-or-wrong answer for at least some inputs, and a fuzzier judgment call for others?

Mix deterministic cases (classification, extraction) that get expected_answer with open-ended cases (summarization, explanation quality) that get a grading_rubric describing what a good answer looks like.

import json
eval_set = [
    {"id": "e1", "input": "Classify sentiment: 'This is fine I guess.'", "expected_answer": "neutral"},
    {"id": "e2", "input": "Classify sentiment: 'Worst purchase of my life, and I've bought a lot of junk.'", "expected_answer": "negative"},
    {"id": "e3", "input": "Summarize this policy doc in 2 sentences: ...", "grading_rubric": "Pass if the summary is <=2 sentences, mentions the refund window, and does not invent details not in the source."}
]
with open("eval_set.json", "w") as f:
    json.dump(eval_set, f, indent=2)

Write an exact-match grader that runs only against the cases with an expected_answer field, comparing the model's output (normalised for case/whitespace) to the expected value.

Exact-match is cheap, deterministic, and unambiguous - the exam expects you to reach for it whenever the task has a single correct answer, rather than defaulting to a model-graded rubric for everything.

You should see: A function that returns a boolean pass/fail per exact-match case, plus an aggregate pass rate printed for just that subset.

Hints

What normalisation would make 'Neutral' and 'neutral ' both count as a correct match without over-loosening the check?

Lowercase and strip both strings before comparing. Keep the comparison strict otherwise - exact match should not silently accept partial matches.

def grade_exact_match(model_output: str, expected: str) -> bool:
    return model_output.strip().lower() == expected.strip().lower()

exact_cases = [c for c in eval_set if "expected_answer" in c]
results = [grade_exact_match(run_model(c["input"]), c["expected_answer"]) for c in exact_cases]
print(f"Exact-match pass rate: {sum(results)}/{len(results)}")

Write an LLM-as-judge grader for the rubric cases: a separate Claude call receives the input, the model's output, and the fixed rubric text, and returns a pass/fail plus a one-line reason. Use a fixed judge prompt you don't tweak per-case.

Open-ended output has no single correct string to match against, so grading falls to a rubric - but the judge prompt itself must stay fixed across every run, or the grading criteria silently drift along with whatever you're testing.

You should see: A function that sends a structured judge prompt to the API and parses a PASS/FAIL verdict plus a reason string out of the response, run against every rubric-graded case.

Hints

What should the judge prompt include so its output is easy to parse programmatically instead of free-form prose?

Ask the judge to answer in a fixed format (e.g. a first line of exactly PASS or FAIL, followed by a one-sentence reason) and parse that first line.

def grade_with_judge(client, input_text, model_output, rubric):
    judge_prompt = (
        f"Input: {input_text}\n\nModel output: {model_output}\n\nRubric: {rubric}\n\n"
        "Respond with exactly PASS or FAIL on the first line, then one sentence explaining why."
    )
    response = client.messages.create(
        model="claude-opus-5", max_tokens=200,
        messages=[{"role": "user", "content": judge_prompt}]
    )
    text = next(b.text for b in response.content if b.type == "text")
    verdict, _, reason = text.partition("\n")
    return verdict.strip().upper() == "PASS", reason.strip()

Run the full harness (exact-match + judge) against the current prompt, then make one deliberate change to the prompt and re-run the identical, unmodified eval_set.json. Store each run's results with model ID, prompt version, and timestamp.

Holding the test set and grading criteria fixed while changing only the prompt is what makes the before/after comparison honest - and storing metadata is what lets you prove later which run produced which score.

You should see: Two result logs (e.g. run_2026-08-29_v1.json and run_2026-08-29_v2.json), each with a per-case pass/fail, an aggregate score, and a metadata block, plus a printed delta including whether the hard edge case passed both times.

Hints

What fields does a later reader need in the results file to know exactly what was tested and how, without re-reading your code?

Include prompt_version, model, dataset_file, dataset_hash (so you can detect if eval_set.json ever silently changed), and a timestamp alongside the per-case results.

import hashlib, json, datetime
def run_harness(client, prompt_version, prompt_text, eval_set):
    dataset_hash = hashlib.sha256(json.dumps(eval_set, sort_keys=True).encode()).hexdigest()[:12]
    results = [run_and_grade(client, prompt_text, case) for case in eval_set]
    log = {
        "prompt_version": prompt_version, "model": "claude-opus-5",
        "dataset_hash": dataset_hash, "timestamp": datetime.datetime.utcnow().isoformat(),
        "results": results, "pass_rate": sum(r["passed"] for r in results) / len(results)
    }
    with open(f"run_{prompt_version}.json", "w") as f:
        json.dump(log, f, indent=2)
    return log

Manually re-grade a random sample of 3-4 judge-scored cases yourself, compare your verdicts to the automated judge's, and flag any disagreement. Then re-run the full harness once more on the winning prompt version and note whether the score moved.

Calibration catches self-preference bias before it silently inflates every future comparison, and a second run on the same version surfaces run-to-run variance so a one-off lucky score doesn't get mistaken for a real improvement.

You should see: A short comparison table (case id, judge verdict, your verdict, agree/disagree) plus a second pass_rate for the same prompt version that you can compare to the first to gauge variance.

Hints

If your manual grade disagrees with the judge on a case, what does that tell you about the rubric prompt versus about the model output itself?

A disagreement usually means the rubric text is ambiguous or the judge is being too lenient/strict in a specific way - tighten the rubric wording rather than discarding LLM-as-judge outright.

sample = random.sample([r for r in results if "judge_reason" in r], k=3)
for r in sample:
    print(r["id"], "judge:", r["passed"], "-", r["judge_reason"])
    my_verdict = input("your verdict PASS/FAIL: ").strip().upper() == "PASS"
    print("AGREE" if my_verdict == r["passed"] else "DISAGREE -- inspect rubric wording")

log_rerun = run_harness(client, "v2-rerun", prompt_v2_text, eval_set)
print("v2 first run:", log_v2["pass_rate"], "v2 second run:", log_rerun["pass_rate"])

Sources

Anthropic docs - Empirical approach to evaluations