Troubleshooting and Optimisation · Lesson 2 of 4

7.2 — Adjusting Approach Based on Feedback and Results

Judge whether a change actually helped by measuring against a consistent set of test cases and criteria, rather than declaring victory after one anecdotally good result.

Diagnosing a problem (lesson 7.1) tells you what to change. It doesn't tell you whether the change actually worked. That second question — did this get better — is only answerable if you're judging "better" the same way before and after the change. This lesson is about the discipline that makes adjustment meaningful: holding your test cases and your criteria for success constant while the thing you're testing changes.

This sounds obvious stated plainly, but it's one of the easiest disciplines to abandon under real deadline pressure, because a single good-looking result feels like enough evidence to move on. It usually isn't.

Why Consistent Measurement Is What Makes Adjustment Meaningful

If you change the test inputs, the grading criteria, or what counts as a "good" output between one version of a prompt or workflow and the next, you can no longer tell whether an apparent improvement reflects the actual change you made or just a difference in how that particular run happened to be judged. Measuring with the same test cases and the same criteria consistently across a comparison is what allows a performance difference to be attributed to the change itself, rather than to a shift in how each version was evaluated. Without that consistency, an apparent improvement — or an apparent regression — might be nothing more than an artifact of testing on an easier input the second time, or grading more leniently because you already expected the new version to be better.

The Trap of One Anecdotally Good Result

A common and tempting failure pattern: you adjust a prompt, run it once against a single input, the output looks noticeably better than you remember the old one being, and you declare the problem solved. This is weak evidence. It's entirely possible the new version happens to do well on that one input and no better — or actually worse — on the range of inputs the workflow needs to handle in practice. A single anecdotal win tells you the change can help; it doesn't tell you the change reliably helps across the cases that matter.

The fix is small and doesn't require formal tooling: keep a short, fixed list of representative test cases (three to five is often enough for a lightweight check) and a simple, consistently-applied rule for what counts as acceptable on each. Run both the old and new version against that same fixed list, applying the same rule, before deciding the adjustment actually helped. If a borderline case comes up mid-comparison, resist the urge to loosen or tighten the rule on the spot — note the ambiguity and apply the same standard to both versions anyway, or the comparison stops being fair.

Key Concept

Before adjusting a prompt or workflow based on feedback, fix a small, consistent set of test cases and a consistent standard for success. Compare the old and new versions against that exact same set. This is what turns "this looks better" into "this is actually better," and it's the difference between a real adjustment and an anecdote.

Common Exam Distractor

An answer suggesting you refresh or vary the test inputs each time you check a new version — "to keep things realistic" — sounds reasonable in isolation but breaks the comparison. Changing what you're testing against between versions makes it impossible to know whether a score changed because the prompt changed or because the test changed. Test-set changes should happen deliberately between full evaluation cycles, not casually mid-comparison. Also watch for answers that treat a single good-looking run as sufficient evidence that an adjustment worked.

Exam traps

Declaring an adjustment successful based on one noticeably better-looking output

A single anecdotal result doesn't establish that a change reliably helps across the range of inputs the workflow actually needs to handle; it only shows the change can help on that one case.
Changing the test cases or grading standard between the old and new version of a prompt

This makes it impossible to tell whether a measured difference reflects the actual change or just a difference in what was tested — a fair comparison requires holding both constant.
Loosening or tightening the success criteria mid-comparison when a borderline case appears

Adjusting the standard partway through biases the comparison toward whichever version you evaluate later; the same standard has to apply to both versions for the comparison to mean anything.

Practice question

After adjusting a prompt to fix a recurring issue, a team runs the new version once against a single input, the output looks clearly better than before, and they conclude the issue is fixed. What is the strongest objection to this conclusion?

A The new prompt is probably longer than the old one, which is inherently worse
Prompt length by itself isn't the issue being raised here — the objection is about the strength of the evidence for improvement, not the prompt's length.
B A single result on one input doesn't establish that the change reliably helps across the range of inputs the workflow needs to handle; the old and new versions should be compared against the same fixed set of test cases and standard Correct
One anecdotal good result is weak evidence — it doesn't rule out the new version doing no better, or worse, on other realistic inputs. A fair, consistent comparison across a fixed test set is needed before concluding the change actually helped.
C The team should have used a different model to generate the comparison output
Model choice isn't the issue raised by this scenario — the problem is the lack of a consistent, multi-case comparison, regardless of which model was used.
D The team should have asked Claude to rate its own output on a scale of 1 to 10 before drawing a conclusion
A single self-rating on the same one input doesn't address the core problem, which is testing against only one case with no fixed comparison standard.

Build exercise: Compare Two Prompt Versions Against a Fixed Test Set

Intermediate · 25 minutes

You'll practice:

Define a small fixed test set and success standard before adjusting a prompt
Run old and new versions against identical test cases and criteria
Recognise how a single anecdotal result would have misled the comparison

Before making any changes, write a short, fixed list of three to five real test inputs for a task you're already using Claude for (e.g. drafting meeting recap emails from notes), along with one simple, consistently-applicable rule for judging each output (e.g. 'does it list every action item mentioned in the notes, yes/no').

Defining the test set and the standard before you change anything is what makes the later comparison fair — you can't retroactively fix a comparison you didn't set up in advance.

You should see: A written list of three to five fixed inputs and a one-line rule you could apply the same way on any output, without ambiguity.
Hints
1. ```
Keep the rule simple enough to check by eye consistently — avoid anything that requires subjective judgement calls that could drift between runs.
```
2. ```
Pick inputs varied enough that a real difference between versions would actually show up.
```
3. ```
Write the list down somewhere you won't be tempted to quietly edit it later.
```
Run your current prompt against all of the fixed test inputs and apply your rule to each output, noting a pass/fail. Then make one targeted adjustment to the prompt and re-run it against the exact same test inputs, applying the exact same rule.

This is the controlled comparison the lesson describes: only the prompt changes between runs; the test cases and standard stay identical.

You should see: Two full sets of pass/fail results, one per prompt version, judged against identical inputs by an identical rule, that you can honestly compare.
Hints
1. ```
Resist adjusting your rule if a borderline case comes up — note it and apply the same call to both versions.
```
2. ```
Count passes for each version rather than relying on a general impression of 'this batch felt better.'
```
3. ```
If the new version doesn't clearly outperform the old one across the full set, that's a real result — don't discard it in favor of the single best-looking case.
```
Pick the single best-looking output from your new version's run and imagine you had only tested that one case. Write one or two sentences on what conclusion you would have wrongly drawn, and how that differs from what the full fixed-set comparison actually showed.

This makes the anecdotal-evidence trap concrete by contrasting it directly against the real comparison you just ran.

You should see: A short written contrast showing that the single best case would have overstated (or understated) how much the adjustment actually helped.
Hints
1. ```
Look specifically for a test case where the new version did no better, or worse, than the old one — that's the case a single-anecdote check would have missed.
```
2. ```
This is the same gap the exam question in this lesson is testing.
```
3. ```
Keep this comparison — it's a useful reference the next time you're tempted to judge a change from one good run.
```

Sources

Reused/adapted from CCAR-F lesson 4-5-consistent-metrics-across-iteration
Anthropic docs — Building Evals