CCDV-F Glossary - Eval, Testing & Debugging

Quick-lookup definitions for every domain, with exam context and links back to the lesson that covers each term.

Error Envelope: The consistent JSON shape returned on every non-2xx Messages API response: a top-level type: 'error', a nested error object with its own type (the precise category, e.g. invalid_request_error) and a human-readable message, plus a request_id. The HTTP status code narrows the category, error.type names it precisely, and the message almost always identifies the exact field or condition at fault.; Exam context: The exam expects you to read the status code, error.type, and message together, not pattern-match on the status code alone. A distractor answer diagnoses a failure from the status code only and ignores the more specific error.type and message text.; See also: 8.1 Debugging Claude API Errors
request_id: A unique identifier returned on every API response, success or failure (in Python, the public response._request_id attribute despite the leading underscore). It should be logged on every failure since it is the first thing Anthropic support asks for and is what lets a user-reported failure be correlated with a specific server-side event.; Exam context: Exam scenarios about debugging a production failure often test whether you know to capture and log request_id, not just the error message, since the message alone can be insufficient to diagnose a server-side issue.; See also: 8.1 Debugging Claude API Errors
Retryable vs Non-Retryable Errors: 429 (rate_limit_error), 5xx (api_error, overloaded_error), and connection errors are transient and retryable with backoff. 400, 401, 403, 404, and 413 are deterministic client-side failures - resending the identical request produces the identical failure every time, so the only fix is changing the request, credential, or model ID before sending again.; Exam context: The exam's central distinction for this lesson. A common trap is catching one broad exception class (e.g. a single except APIStatusError) and treating every failure identically, which discards exactly this retryable/non-retryable split. Also watch for answers that retry a 400/401/403/404 unchanged, expecting a different outcome.; See also: 8.1 Debugging Claude API Errors
401 Authentication Error vs 403 Permission Error: A 401 authentication_error means the request never proved who it is - the API key is missing, malformed, or revoked. A 403 permission_error means the request's identity was proven and is valid, but that identity isn't allowed to do the specific thing it asked for (call a model it hasn't been granted, use a beta feature, touch an out-of-org resource). Neither is retryable without fixing credentials (401) or access (403).; Exam context: A classic exam trap treats a 403 as equivalent to a 401 and 'fixes' it by rotating or re-checking the API key - that does nothing, since authentication already succeeded. Seeing a 403 on a model that worked minutes earlier signals an access change, not a bad key. Note both permission_error and billing_error can also surface as 403.; See also: 8.1 Debugging Claude API Errors
429 Rate Limit Error vs 529 Overloaded Error: Both are retryable and both mean 'try again later,' but the cause differs. A 429 rate_limit_error means *your account* exceeded its own requests-per-minute, tokens-per-minute, or tokens-per-day allowance - check x-ratelimit-remaining-* and retry-after headers; the durable fix is pacing your request rate. A 529 overloaded_error (or a 500) means Anthropic's infrastructure is capacity-constrained or has a transient fault, entirely independent of your usage - there's no account-side fix, only backoff.; Exam context: Expect a scenario asking what a 529 specifically indicates as distinct from a 429; the answer hinges on whose capacity is the bottleneck (Anthropic's vs. your account's), not on retryability, since both are retryable.; See also: 8.1 Debugging Claude API Errors
Typed Exception Hierarchy: Each SDK maps every status code to its own exception class (Python: BadRequestError, AuthenticationError, PermissionDeniedError, NotFoundError, RateLimitError, InternalServerError), all subclasses of APIStatusError, with APIConnectionError covering network failures before any response arrives. Catching should go most-specific-first, since a general clause listed before a specific subclass silently swallows it.; Exam context: The exam tests correct except-clause ordering: RateLimitError (a subclass of APIStatusError) must be caught before the general APIStatusError clause, or it never gets reached. A single broad except APIStatusError (or a bare HTTP status check) that handles every failure the same way is the standard wrong answer.; See also: 8.1 Debugging Claude API Errors
Golden Dataset: A fixed, representative set of inputs - real or realistic, including known-hard edge cases - each paired with an expected answer (for exact-match tasks) or a grading rubric (for open-ended ones). It replaces ad hoc manual spot-checking, which only confirms the handful of cases someone happened to try. When a real production failure occurs, the fix includes adding that exact case to the dataset permanently, so a future change can't silently reintroduce it.; Exam context: A trap is building the dataset only from cases the current prompt already handles well, with no deliberately hard or known-failure cases - that produces a high pass rate regardless of real quality and can never catch a regression. A dataset that always scores 100% has stopped telling you anything.; See also: 8.2 Building an Evaluation Harness
Grading Method: Exact-Match vs Rubric-Based: Exact-match or programmatic checks (a classification label, an extracted JSON field, a regex, a numeric value within tolerance) should be preferred whenever the task has one correct answer - they're cheap, deterministic, and unambiguous. For open-ended output (summary quality, tone, whether instructions were followed), exact match doesn't apply, so grading falls to human raters or a model-graded rubric (LLM-as-judge).; Exam context: The exam expects you to default to exact-match whenever a task allows it, rather than reaching for a fuzzier LLM-as-judge grade by default. Recognizing which of a task's cases are deterministic vs. open-ended is the tested skill.; See also: 8.2 Building an Evaluation Harness
LLM-as-Judge: A grading pattern where a separate Claude call receives the original input, the output being graded, and a fixed rubric, then returns a score (pass/fail or 1–5) plus a short justification. It scales far better than human grading and can catch nuance exact-match can't express, but it must use a fixed rubric prompt held constant across comparisons.; Exam context: Exam scenarios test whether you know LLM-as-judge is legitimate but not automatically trustworthy on its own - it needs a fixed rubric and periodic human calibration (see Self-Preference Bias) to be a valid grading method, not something to avoid entirely.; See also: 8.2 Building an Evaluation Harness
Self-Preference Bias: The risk that a model grading outputs from its own model family - especially its own kind of output - rates them more favorably than an independent judge would. The structural mitigations: hold the rubric prompt fixed across comparisons, prefer a different or stronger model as judge than the one under test, and periodically sample automated grades for a human to re-check.; Exam context: A common distractor uses a model-graded rubric with no fixed rubric prompt and no periodic human check, treating 'model-graded' as automatically trustworthy. The exam wants the mitigation (fixed rubric + human spot-checks), not abandoning LLM-as-judge altogether.; See also: 8.2 Building an Evaluation Harness
One-Variable-at-a-Time Comparison: For a before/after eval comparison to mean anything, the test set and grading criteria must stay fixed while exactly one thing changes - the prompt, the model, or a single parameter. Every run's results should be stored with full metadata (model ID, prompt version/hash, dataset version, timestamp) so any two runs can be diffed unambiguously later; a dataset update is logged as its own deliberate event, never folded silently into an unrelated change.; Exam context: The signature exam trap: changing the eval test set at the same time as the prompt, then comparing scores - the difference can't be attributed to either change specifically. This is a more fundamental problem than small sample size, even when both are present in a scenario.; See also: 8.2 Building an Evaluation Harness
Run-to-Run Variance: Claude's output is not perfectly deterministic even with identical inputs and default settings, so a score can shift by a few points between two runs of the exact same prompt against the exact same dataset. Larger datasets average out individual-example volatility, and running each version multiple times and comparing distributions (not single scores) is more reliable than one run.; Exam context: The exam tests recognizing that a single run on a small sample (e.g. a jump from 80% to 88% on ten examples) is not sufficient evidence to declare a regression fixed or a real improvement - it could be noise from one or two examples flipping.; See also: 8.2 Building an Evaluation Harness