HTTP Error Code Reference
| Status | error.type | Cause | Retryable? | Fix |
|---|---|---|---|---|
| 400 | invalid_request_error | Malformed request: bad JSON, missing required param, invalid tool schema, non-alternating roles | No | Fix the request itself |
| 401 | authentication_error | Missing, invalid, or revoked API key (or both ANTHROPIC_API_KEY and ANTHROPIC_AUTH_TOKEN set at once) | No | Fix credentials |
| 403 | permission_error | Key is valid but lacks access to this model, beta feature, or org resource | No | Grant access or use a different key |
| 404 | not_found_error | Bad endpoint or invalid/deprecated model ID (e.g. dotted claude-sonnet-4.6 instead of claude-sonnet-4-6) | No | Fix the model ID or endpoint |
| 413 | request_too_large | Request body exceeds size limit - oversized images or unbounded history | No | Shrink the payload |
| 429 | rate_limit_error | Your account exceeded its RPM/TPM/TPD limit | Yes | Back off; honor retry-after |
| 500 | api_error | Transient problem on Anthropic's side | Yes | Backoff and retry |
| 529 | overloaded_error | Anthropic's infrastructure at capacity (not your usage) | Yes | Backoff; consider routing elsewhere |
Read the status code, error.type, and message together - the message almost always names the exact field or condition at fault. Always log request_id on failures.
SDK Auto-Retry Behavior
- SDKs auto-retry connection errors, 408, 409, 429, and any 5xx with exponential backoff + jitter, up to
max_retries(default 2, configurable). - A 429 with a
retry-afterheader is honored exactly instead of a guessed delay. - 400, 401, 403, 404, 413 are never auto-retried - they're deterministic; resending identical bytes reproduces the identical failure.
- Don't wrap a custom retry loop around the SDK's own retrying - that produces a multiplicative retry storm on a single failure.
- A 404 fails almost instantly (no retry delay); a retried 429/5xx shows multi-second backoff delays - useful for confirming which path fired.
Telling Similar Codes Apart
| Pair | Distinction | Distractor to Reject |
|---|---|---|
| 401 vs 403 | 401 = identity never proven (bad/missing key). 403 = identity proven, but not authorized for this resource. | 'Fixing' a 403 by rotating the API key - the key is already fine |
| 429 vs 529 | 429 = your account's own rate limit. 529 = Anthropic's infrastructure is overloaded, unrelated to your usage. | Treating both as the same 'your fault, slow down' signal |
| 400 on a tool-calling request | Schema validation fails before the model ever runs - nothing about model behavior is implicated. | Re-prompting or blaming the model's reasoning for a schema 400 |
Exception catch order (Python): most specific first - NotFoundError → RateLimitError → APIStatusError → APIConnectionError. Since RateLimitError subclasses APIStatusError, put it first or the general clause swallows it silently.
Evaluation Harness: The Three Components
- Golden dataset - representative inputs incl. known-hard edge cases, each with an expected answer or grading rubric. Version it like code; never edit it in the same commit as the thing it tests.
- Grading method - exact-match/programmatic where the task has one right answer; human or LLM-as-judge rubric where it doesn't.
- Repeatable runner - same dataset + same grading logic every run, aggregated score + per-example pass/fail, written to a versioned log.
- A dataset that always scores 100% has stopped telling you anything - deliberately include cases you're not confident will pass.
Grading Method Comparison
| Method | Use When | Watch Out For |
|---|---|---|
| Exact-match / programmatic | One correct answer: classification label, extracted field, regex, numeric tolerance | Default to this whenever the task allows it - don't reach for a rubric unnecessarily |
| Human rating | Open-ended output, high-stakes or low-volume review | Doesn't scale; use for calibration spot-checks even when automated grading is primary |
| LLM-as-judge (rubric) | Open-ended output at scale: summary quality, tone, instruction-following | Self-preference bias if judge shares a model family with the model under test; needs a fixed rubric prompt + periodic human re-check |
Before/After Comparison: Do / Don't
| Do | Don't |
|---|---|
| Hold the test set and grading criteria fixed; change exactly one variable (prompt, model, or parameter) | Change the dataset and the prompt in the same comparison - you lose the ability to attribute the score difference |
| Log model ID, prompt version/hash, dataset version, and timestamp with every run | Rely on memory of how 'the last version' scored |
| Use a stronger/different model as judge than the one under test; fix the rubric prompt | Let a model grade its own family's output with no fixed rubric or human check |
| Run multiple times / use a larger dataset and compare distributions before declaring a win | Declare an improvement (or a fixed regression) from one run on a small sample - a few points can be noise |
| Include deliberately hard/known-failure cases in the dataset | Build the dataset only from cases the current prompt already passes |