Evaluation, Testing & Optimisation · Lesson 4 of 6

3.4 - Diagnosing Failures: Prompt Failure, Hallucination and Model Mismatch

Work through a layered diagnostic tree to decide whether a bad output comes from the API call, the tools, the retrieved context, the prompt, or the model, and collect the right evidence before changing anything.

“The answers are bad” is a symptom, not a diagnosis. The most expensive mistake in a Claude system is fixing the wrong layer: rewriting a prompt when retrieval never surfaced the document, upgrading the model when a tool schema was rejected, or blaming hallucination on the generator when the source was never in context. The exam scenarios in this area usually give you a few observations and ask which layer is actually at fault, or which single step you should take next. The skill is to hold hypotheses in order of cost and to pick the test that separates them.

Before touching the prompt, collect evidence. For each failing case you want: the request-id of the response (every API response carries one; the Python and TypeScript SDKs expose it as _request_id); the full request as actually sent, including system prompt, tool definitions, message history, model ID and parameters such as effort and max_tokens; the full response including stop_reason and usage; the tool calls and tool results; the retrieved chunks with identifiers and scores; and the prompt version. Then reproduce the failure, reduce it to the smallest failing input, and only then choose a hypothesis. Lesson 1.4 covers how to capture this at scale; Lesson 7.3 covers the operational side in Claude Code environments.

The diagnostic tree, cheapest layer first

Did the call succeed and finish? Read the status, the error.type and the message together. A 400 invalid_request_error means the request was malformed and the model never ran, so nothing about model behaviour is implicated; a tool-call 400 is often a schema problem. 401 is a bad credential, 403 is a valid credential lacking access to that resource, 404 is often a wrong model ID, and 413 is an oversize request. These are deterministic: fix the request, do not resend it. 429 (a rate limit), 500 and 529 (overload) are the transient ones, and the official SDKs already retry transient failures with backoff, twice by default. A 429 caused by a tier spend cap is the exception: it has no retry-after header and keeps failing until access resumes, so it needs an alert, not a retry. Also check for truncation (stop_reason of max_tokens means the answer was cut off) and note that a streaming response can fail after a 200 has already been returned.
Did the tools and integration behave? Was the right tool chosen, with valid arguments? Did the tool return an error that was swallowed, or a result that was truncated or malformed? Overlapping or bloated tool sets degrade selection (Lesson 1.1).
Was the needed information in the context? Test by pasting the known-correct source into the prompt. If the answer becomes right, the failure is retrieval or context assembly (chunking, index, query, freshness, trimming or cleared tool results), not the prompt or the model. See Lessons 1.5, 1.6 and 6.4.
Is the prompt the problem? With correct context and a working call, look for ambiguous or conflicting instructions, missing criteria, an output format that is described but not shown, or an instruction buried among many. Iterate in small steps: name precisely what is wrong with a real failing output, make one targeted change, and re-run. You can ask Claude itself to diagnose, but give it the prompt and a concrete failing example; “make this better” with no example only produces guesses.
Is the model or its configuration mismatched to the task? If the same prompt and gold context still fail, run the same case at a higher effort setting or on a stronger tier. If that fixes it, you have a capability or effort mismatch; if it does not, look upstream again.

Key concept: isolate by ablation

Change one layer at a time and hold the rest fixed: gold context instead of retrieved context, a stronger model instead of the current one, a targeted prompt edit instead of a rewrite. The layer whose substitution fixes the failure is the layer at fault. Each ablation is a small eval, so run it on a handful of failing cases, not one.

Hallucination: absent evidence or unfaithful use of evidence

A hallucination is plausible output that is not supported by the source or the facts. Treat it as a symptom with two very different causes.

The truth was not available. Retrieval missed the document, the document was never indexed, or the context was trimmed. The model fills the gap with something plausible. The fix belongs to retrieval and to permission to abstain: Anthropic’s guidance is to explicitly allow Claude to say it does not know.
The truth was available but not used faithfully. The claim contradicts or goes beyond the provided text. Here Anthropic’s techniques apply: for long documents ask Claude to extract word-for-word quotes first and answer only from them; have it cite a supporting quote for each claim and retract claims it cannot support; restrict it to the provided documents rather than general knowledge; ask for step-by-step reasoning before the answer; compare several runs, since inconsistency across outputs can signal fabrication.

The separating test is a groundedness check: does the wrong claim appear anywhere in the context that was sent? Anthropic’s own caveat is that these techniques reduce hallucination but do not eliminate it, so validate critical information for high-stakes use, and add a verification layer where consequences are serious (Lessons 4.2 and 4.3).

Model mismatch: capability, configuration and migration

Suspect the model when failures are consistent across reasonable prompt variants, when correct context is present, and when a stronger tier or higher effort setting fixes them. Anthropic’s guidance is to establish criteria, build evaluation sets specific to your use case, and compare models on accuracy, quality and edge-case handling rather than assuming; it also notes that tuning effort is often a better lever than switching models. Three mismatch patterns are worth recognising:

Effort or tier too low for the task. Lower effort trades some capability for speed and cost, so complex multi-step reasoning can degrade. Set effort explicitly rather than relying on defaults.
Migration breakage. A prompt tuned for one model may behave differently on another. The docs list concrete API-level examples: assistant-message prefill is rejected with a 400 on newer models, and the manual extended-thinking parameter is rejected on the newest ones in favour of adaptive thinking with effort. These show up as errors, not as quality drops, so read the message before blaming the prompt.
Token accounting shifts. The docs note that newer tokenizers produce noticeably more tokens for the same text, so prompts near a context or cost limit can behave differently after a migration. Re-count against the target model.

Common exam distractor

The tempting answers are the ones that skip diagnosis: rewrite the whole prompt after one bad output, switch to the largest model, add “do not hallucinate” to the system prompt, or retry a 4xx request unchanged. Each fixes a layer you have not shown to be broken. The correct answer normally names the layer the evidence points to and the smallest test or change that targets it.

From single failures to a failure profile

One failure tells you where to look; a sample tells you where to invest. Label 20 to 50 real failures by root cause (call error, tool, retrieval, prompt, model, data or label problem), tally them, and fix the largest bucket first. Each fixed failure then becomes a permanent case in the evaluation set (Lesson 3.2) so it cannot return silently, and a disagreement between your labels and an automated grader is itself a finding about the grader.

Exam traps

After one poor output, discarding the prompt and rewriting it from scratch (or asking Claude to 'make it better' with no example).

Without a named failure and a concrete failing example there is nothing to diagnose against. A prompt that mostly works needs a targeted change verified against real failures, not a replacement that risks breaking what works.
Treating a 400 invalid_request_error, 401, 403 or 404 as a model-quality problem and retrying or switching models.

These are deterministic request-side failures: the model never ran. Resending the same bytes gives the same result. Read the error type and message, then fix the request, credential, permission or model ID. Only transient failures (a rate-limit 429 with backoff, 5xx and 529 responses, connection errors) are worth retrying, and a spend-cap 429 is not one of them.
Assuming a hallucination always means the generating model is too weak, and upgrading the tier as the first response.

If the correct information was never retrieved, or was trimmed from context, a stronger model will still fill the gap. Check whether the truth was in context first; then use grounding techniques (allow 'I don't know', quotes, citations, restrict to provided documents).
Adding an instruction such as 'never make things up' to the prompt as the complete fix for hallucination.

Such instructions help at the margin, but Anthropic notes that mitigation techniques reduce hallucination without eliminating it. Diagnosing whether the failure is absent evidence or unfaithful use, and adding grounding and verification, targets the actual cause.

Practice question

A retrieval-augmented support assistant tells a customer that returns are accepted for a period that contradicts the current policy. The team's request logs show the retrieved chunks for that query were about shipping and warranty, and the returns-policy section is absent. The prompt and tool calls are otherwise fine, and the API call succeeded. What is the most likely root cause and best next step?

A The model tier is too weak for policy questions; move the assistant to the most capable tier, re-run the failing query and adopt the new tier if the answer improves, accepting the higher per-token cost.
A stronger model given the wrong context will still lack the returns policy. The evidence shows the needed information never reached the prompt, so the model is not the first suspect.
B A retrieval failure: the policy section was not surfaced. Check chunking, indexing and query handling, confirm with a gold-context test that supplying the policy fixes the answer, and add the query to the eval set. Correct
The logs point to context assembly. The gold-context test separates retrieval from prompt or model, and adding the case to the eval set prevents a silent recurrence.
C The prompt needs a stronger instruction never to state policies it is unsure about; add it, redeploy and monitor whether the assistant stops giving wrong return periods to customers over the next few days.
Permission to abstain helps when evidence is missing, but it treats a symptom. The concrete fault is that retrieval omitted the relevant section, and a prompt change would leave that unaddressed.
D Raise the effort setting to maximum so the model reasons harder about the policy question and cross-checks the returns period before answering, then compare the new answers with the current ones.
More reasoning cannot recover facts that are absent from the context. Effort helps when a task exceeds the model's capability with adequate context, which the evidence does not show.

Build exercise: Build a failure-triage kit: evidence capture, layer ablations and a failure profile

Advanced · 75 minutes

You'll practice:

Capture a complete evidence bundle (request id, request, stop reason, usage, context) for every call
Use gold-context and model or effort ablations to isolate which layer is failing
Distinguish absent-evidence hallucination from unfaithful-use hallucination with a groundedness check
Tally labelled failures into a profile that prioritises fixes and feeds the evaluation set

Write a wrapper around your Claude call that appends one JSON line per call to a log with: prompt version, model, the request parameters, request id, stop_reason, usage, and on failure the HTTP status and error message. Do not log secrets, and note where customer data could appear in the log.

You cannot diagnose from memory. A consistent evidence bundle is what lets you reproduce a failure, hand it to support with a request id, and compare runs.

You should see: A log file where a successful call and a deliberately failing call (for example a fake model ID) each produce a complete record, with the failing one showing a 404 and its request id.

Hints

If support asks for the request id of a failed call, where would you look, and how does that differ between a success and an error?

Read the id from the response object on success and from the error's response headers on failure; record stop_reason and usage on success; catch the SDK's status-error class for failures.

import json, time, anthropic
client = anthropic.Anthropic()

def call_logged(log_path, prompt_version, **request):
    rec = {"ts": time.time(), "prompt_version": prompt_version,
           "model": request.get("model"), "max_tokens": request.get("max_tokens")}
    try:
        resp = client.messages.create(**request)
        rec.update(ok=True, request_id=resp._request_id, stop_reason=resp.stop_reason,
                   usage=resp.usage.model_dump())
        return resp
    except anthropic.APIStatusError as e:
        rec.update(ok=False, status=e.status_code, message=str(e.message),
                   request_id=e.response.headers.get("request-id"))
        raise
    finally:
        with open(log_path, "a", encoding="utf-8") as f:
            f.write(json.dumps(rec) + "\n")

Take five failing questions from a retrieval-based assistant (or build a small one over a few documents). For each, run the system twice: once with the context your retriever returned, and once with the known-correct source pasted in. Record which failures are fixed by gold context.

This one ablation separates retrieval or context problems from prompt and model problems, which is the most common misdiagnosis.

You should see: A table of five cases with retrieved-context result, gold-context result and a verdict: retrieval or context fault if gold context fixes it, otherwise continue down the tree.

Hints

For which failing cases would you expect gold context to make no difference, and what would that tell you?

Keep the prompt, model and parameters identical between the two runs; only the context block changes. Check the answer against the ground truth with the same grader.

def answer(question, context, model="claude-sonnet-5"):
    r = client.messages.create(
        model=model, max_tokens=500,
        system="Answer only from the provided context. If it is insufficient, say you do not know.",
        messages=[{"role": "user", "content": f"<context>{context}</context>\nQuestion: {question}"}])
    return next(b.text for b in r.content if b.type == "text")

for case in failing_cases:
    got_retrieved = grade(answer(case["q"], case["retrieved"]), case["truth"])
    got_gold = grade(answer(case["q"], case["gold"]), case["truth"])
    print(case["id"], "retrieved:", got_retrieved, "gold:", got_gold)

For the failures that still fail with gold context, run the same prompt across a small grid of configurations: the current model at its current effort, the same model at a higher effort, and the next tier up. Confirm the model IDs and effort support on the current models page before you start. Record which configuration, if any, fixes each failure.

If only a stronger configuration fixes it, you have a capability or effort mismatch and the cost trade-off becomes an architectural decision. If nothing fixes it, the fault is upstream in the prompt or the labels.

You should see: A grid of configuration by case with pass or fail, and for each case a conclusion: capability or effort mismatch, prompt issue, or bad label.

Hints

If a case passes only at the highest effort on the strongest model, is that necessarily the right fix once cost and latency are considered?

Sweep effort on the current model first (cheapest experiment), then a stronger tier, and price the winner per completed task before adopting it.

configs = [("claude-sonnet-5", "low"), ("claude-sonnet-5", "high"), ("claude-opus-5", "high")]  # verify on the models page
for model, effort in configs:
    r = client.messages.create(
        model=model, max_tokens=2000, output_config={"effort": effort},
        system="Answer only from the provided context. If it is insufficient, say you do not know.",
        messages=[{"role": "user", "content": f"<context>{case['gold']}</context>\nQuestion: {case['q']}"}])
    print(model, effort, grade(next(b.text for b in r.content if b.type == "text"), case["truth"]))

Pick three answers that contain claims you believe are hallucinated. For each, search the context that was sent for the claim. Classify it as absent evidence (claim not in context, and the truth was also missing) or unfaithful use (truth was in context but the answer contradicted or exceeded it). Then apply one matching technique and re-run.

The two causes call for different fixes, so the classification decides whether you change retrieval or the grounding instructions.

You should see: Three classified cases and, for each, a before and after result after a single matching change (for example quote-first extraction or an explicit permission to say the context is insufficient).
Hints
1. ```
If the claim appears nowhere in the context, could the model possibly have got it from anything other than its general knowledge?
```
2. ```
Absent evidence: fix retrieval and allow abstaining. Unfaithful use: extract quotes first, require a supporting quote per claim, restrict to the provided documents.
```
3. ```
Example grounding instruction: First extract the word-for-word quotes from the documents that are relevant to the question and list them. Then answer using only those quotes, citing each by number. If no quote supports a claim, do not make the claim. If the documents are insufficient, say: I do not have enough information to answer.
```
Label 20 or more failures from your logs with a root-cause tag (call error, tool, retrieval, prompt, model mismatch, label or data problem) and tally them. Write a one-page decision record: which bucket you will fix first, the ablation that supports it, the change, and the evaluation cases you will add so the failure cannot return.

A failure profile turns firefighting into prioritisation and leaves a permanent regression test behind each fix.

You should see: A tally by root cause, a decision record that names the largest bucket and its supporting evidence, and a list of new eval cases with their slice tags.
Hints
1. ```
Which bucket would you fix first if it were the largest but also the cheapest to fix, and does that ordering change if one bucket is safety-relevant?
```
2. ```
Count per tag, sort descending, and weight by severity where you have it. Tie each proposed change to the ablation that showed the layer was at fault.
```
3. ```
from collections import Counter
labels = [f["root_cause"] for f in labelled_failures]   # e.g. "retrieval", "prompt", "model", "tool", "call_error", "label"
for cause, n in Counter(labels).most_common():
    print(f"{cause:12s} {n:3d}  ({n/len(labels):.0%})")
```

Sources

Reduce hallucinations (Claude docs) - Anthropic
Claude API errors (Claude docs) - Anthropic
Choosing the right model (Claude docs) - Anthropic
Effort (Claude docs) - Anthropic
Increase output consistency (Claude docs) - Anthropic
Token counting (Claude docs) - Anthropic