Applications & Integration · Lesson 10 of 11

1.10 - Testing Claude-Integrated Applications

Apply ordinary software-engineering testing discipline to code that calls a non-deterministic model, without pretending the model is deterministic.

The parts of a Claude-integrated feature that are deterministic — request construction, response parsing, tool execution, error handling, the retry and caching logic from earlier lessons — are unit-testable exactly like any other code, and should be, independent of the model itself. Mock the API boundary for these tests: assert your code builds the right request (correct system placement, correct tool_choice, correct cache_control position) and handles a given canned response correctly (correct block-type dispatch, correct tool_use_id matching, correct handling of an unexpected stop_reason), without making a real network call for every test run. Fast, deterministic, and they run in CI on every commit.

Testing the model's behaviour, not just your code

Whether Claude's actual output meets a quality bar is a different kind of test — not a pass/fail unit test but an evaluation against a curated set of representative inputs with a grading rubric (Domain 8 goes deeper on building that harness). Conflating the two — asserting exact string equality on live model output in a CI unit test — produces a flaky, low-value test that breaks on any harmless phrasing change, forcing a developer to either loosen the assertion until it's meaningless or babysit the test every time the wording shifts.

What to assert on non-deterministic output

When a test does need to touch live or recorded model output, assert structural and semantic properties instead of exact text: does the response contain a valid JSON object matching the expected schema, does a required field exist and pass a type check, does the answer contain a specific fact or number regardless of surrounding phrasing, does a classification land in the expected category. This is a weaker but far more durable assertion — it survives the model rephrasing an answer while still catching a genuine regression, like a missing required field or a wrong category.

Common exam distractor

Asserting an exact string match against live Claude output in an automated test suite is a trap answer. Non-deterministic generation means the same prompt can validly produce differently-worded output; test structure and correctness properties, not exact wording.

Recorded fixtures as a middle ground

A useful middle ground between a fully mocked unit test and a live model call on every CI run is a recorded fixture: capture a real response once, save it as a canned payload, and replay it against your parsing/handling code in fast, deterministic tests. This tests your code's handling of a realistic response shape without paying for or waiting on a live call every run, though it's worth periodically refreshing fixtures so they don't drift silently out of sync with what the live API actually returns as models and response shapes evolve.

Evaluation harnesses and LLM-as-judge

A quality evaluation needs a way to score an open-ended answer at scale, and one common approach is using a separate Claude call as a grader — feeding it the original prompt, the candidate response, and a rubric, and asking it to score or pass/fail the response against that rubric. This scales far better than manual review of every case, but it introduces its own reliability question: the grading prompt itself needs validation (does it agree with a human's judgment on a sample of cases?), and it's worth remembering the grader is itself a non-deterministic model call, which is why eval scores are usually reported as a pass rate across many cases rather than trusted on any single case in isolation.

Curating a golden dataset

An eval set's value depends heavily on what it covers: a handful of easy, typical-case prompts will pass cleanly on almost any reasonable model configuration and catch nothing. A useful golden dataset deliberately includes edge cases the application has actually failed on before (regression cases), ambiguous inputs where the right behaviour is a judgment call, and adversarial or malformed inputs the feature needs to handle gracefully rather than crash on. Building this set is itself an ongoing process — every real production failure is a candidate to add as a new eval case, so the dataset grows to reflect the actual failure modes the application has encountered rather than staying frozen at whatever the team thought to test on day one.

Key concept

A one-line rule of thumb the exam rewards: if a test needs to be re-recorded or hand-tuned every time the model's wording changes, it's testing the wrong thing. Structural and behavioural properties survive rephrasing; exact text doesn't.

Exam traps

Writing a CI test that asserts response.content[0].text equals an exact hardcoded string from a live model call.

Model output for a given prompt isn't guaranteed to be byte-identical across calls, so an exact-string CI assertion is flaky by construction - test for structural or semantic properties instead.
Making every real Claude call inside unit tests instead of mocking the API boundary for deterministic code paths.

Request construction, response parsing, and error handling are deterministic code and should be tested with mocked responses - a live network call on every test run is slow, costly, and introduces model non-determinism into a test that's really checking your own code.
Treating an evaluation against a curated dataset with a grading rubric as equivalent to a pass/fail unit test.

Model quality evaluation is a different kind of test than a deterministic unit test - it measures a distribution of outcomes against a rubric, not a single binary pass/fail assertion, and conflating the two produces either flaky unit tests or an underpowered evaluation.
Never refreshing recorded response fixtures used in tests, treating them as permanently accurate.

Fixtures can drift out of sync with what the live API actually returns as models and response shapes evolve over time - periodically refreshing them against a real call keeps fixture-based tests honest.

Practice question

A developer wants CI to catch it if a code change accidentally breaks the request payload sent to the Messages API. What's the appropriate test design?

A A unit test that mocks the API client and asserts the constructed request has the expected fields, with no live network call. Correct
Request construction is deterministic code - a mocked unit test catches a broken payload reliably, quickly, and without depending on the live API or a non-deterministic model response.
B A live call to the API on every CI run, asserting the response text matches a fixed string exactly.
This tests non-deterministic model output with a deterministic assertion - it will be flaky, slow, and expensive, and it's not even testing the thing being asked about (the request payload).
C Manual testing only, since API request construction can't be automated.
Request construction is ordinary deterministic code and is entirely automatable with standard unit-testing practice.
D A test that checks the API's uptime status page before each CI run.
Uptime has nothing to do with whether the code under test constructs a correct request payload.

Build exercise: Write mocked unit tests for request construction and response handling, plus one structural assertion on live output

Intermediate · 35 minutes

You'll practice:

Mock the API boundary in a test
Separate deterministic code tests from model-output evaluation
Write a structural assertion that survives non-deterministic phrasing

Take a function in your codebase that builds a Messages API request, and write a unit test that calls it directly (no network call) and asserts on the resulting object's fields.

This isolates and locks down the one part of the system that genuinely is deterministic and worth a strict assertion.

You should see: The test passes without making any real HTTP call and fails if you deliberately break a field name in the request-building code.

Hints

If your request-building logic is tangled together with the actual network call, what would make both easier to test?

Separate the pure function that builds the request object from the function that actually sends it - the pure function is what you unit test directly, with no mocking needed at all.

test("builds request with system as top-level field", () => {
  const req = buildMessagesRequest({ system: "Be concise.", userText: "Hi" });
  expect(req.system).toBe("Be concise.");
  expect(req.messages).toEqual([{ role: "user", content: "Hi" }]);
});

Write a second unit test that feeds a canned, mocked API response (including a tool_use block and a text block together) into your response-handling code, and assert it dispatches each block correctly.

This locks down the block-type dispatch logic from Lesson 1.1 without depending on the model actually returning that shape on demand.

You should see: The test passes with a hand-constructed mock response object, and fails if you revert the dispatch logic to a fixed-index content[0] read.

Hints

How would you construct a fake response object that has both a text block and a tool_use block, without calling the API at all?

Hand-write a JSON object matching the real response shape - id, content array with both block types, stop_reason - and pass it straight into your handler function as if it came from the SDK.

const mockResponse = { content: [
  { type: "text", text: "Let me check that." },
  { type: "tool_use", id: "toolu_1", name: "lookup", input: { id: 42 } }
], stop_reason: "tool_use" };
const dispatched = handleResponse(mockResponse);
expect(dispatched.toolCalls).toHaveLength(1);

Make one real (or recorded-fixture) call to Claude asking it to classify a short piece of text into one of three fixed categories, and write a test that asserts the result is one of those three categories - not an exact string match on any surrounding explanation.

This is the structural-assertion pattern the lesson argues for: durable against rephrasing, still catches a genuine regression like an invalid category.

You should see: A passing test that would still pass if Claude phrased its reasoning differently, but would fail if the returned category were outside the allowed set.
Hints
1. ```
What would break this test if the model rephrased its answer, versus what should break it if the model's classification is genuinely wrong?
```
2. ```
Force a tool call with an enum-constrained field for the category, then assert only that the returned value is a member of the allowed set - ignore any free text alongside it.
```
3. ```
const allowed = ["billing", "technical", "other"];
const result = await classify("My card was charged twice");
expect(allowed).toContain(result.category); // not: expect(result.text).toBe("...")
```

Sources

Anthropic docs - Best practices for testing
Anthropic docs - Evaluating prompts