Prompt & Context Engineering · Lesson 2 of 4

4.2 - Structured Output and Validation Loops

Force reliable structured output with tool schemas, then add a validation/retry loop for the cases that still slip through.

The reliable way to get structured output from Claude is a tool call constrained by a JSON Schema input_schema, not asking for JSON in free text and parsing whatever comes back. Define a tool (name, description, input_schema), force the model to call it, and read the arguments straight off the tool_use content block — already a parsed object, not a string you have to regex out of a paragraph of preamble. Prompt-based JSON ("respond with a JSON object containing...") gives no structural guarantee at all: a stray sentence before the {, a trailing comma, or an unquoted key will periodically break a plain JSON.parse in production, and the failure rate goes up, not down, as your schema gets more complex.

tool_choice: auto, any, and a named tool

The tool_choice parameter is the actual lever, and it has three distinct modes:

{"type": "auto"} (the default) — the model decides whether to call a tool at all. It can legitimately return plain text instead. Fine for a conversational agent; wrong whenever you need guaranteed structured output, because "guaranteed" and "the model may choose not to" are contradictory.
{"type": "any"} — the model must call some tool, but picks which one. Use this with multiple candidate schemas (extract_invoice, extract_receipt, extract_contract) when the input type is unknown ahead of time: structured output is guaranteed, tool selection stays flexible.
{"type": "tool", "name": "extract_metadata"} — the model must call this specific tool. Use it to force a mandatory step (e.g. metadata extraction must run before an enrichment step), with zero flexibility.

const res = await client.messages.create({
  model: "claude-sonnet-5",
  max_tokens: 1024,
  tool_choice: { type: "tool", name: "extract_invoice" },
  tools: [extractInvoiceTool],
  messages: [{ role: "user", content: documentText }],
});
const data = res.content.find(b => b.type === "tool_use").input;
// data is already a parsed object matching input_schema

What a schema does not guarantee

A schema constrains shape, not correctness. tool_use with a JSON Schema eliminates syntax errors — the arguments will always parse, always have the right types, always satisfy the enum/required constraints you declared. It does not stop the model from producing a field that's well-formed and still wrong: line items that don't sum to the stated total, a date parsed into the wrong field, or — the most damaging case — a plausible-looking fabricated value for a required field the source document never actually mentioned. Required fields specifically pressure fabrication: if the schema says a field must be present, the model is structurally incentivised to invent something rather than fail the call.

The schema-design fix is to make fields optional/nullable whenever the source may legitimately lack that information, so the honest answer (null) is available instead of forcing an invention:

{
  "type": "object",
  "properties": {
    "invoice_number": { "type": "string" },
    "vendor_name": { "type": "string" },
    "payment_terms": { "type": ["string", "null"] },
    "status": { "type": "string", "enum": ["paid", "unpaid", "overdue", "unclear"] }
  },
  "required": ["invoice_number", "vendor_name", "status"]
}

Note the "unclear" value alongside the real enum options — it gives the model a legitimate way to express genuine ambiguity in a constrained field instead of forcing a guess between the real options. An "other" value paired with a freeform *_detail string field does the same job for open-ended categorisation.

Closing the gap: validation and bounded retry

Semantic validation is your code's job, not the schema's. After receiving the tool call, check the values your schema couldn't: does subtotal + tax == total? Is the enum value one your business logic actually expects, or a legitimately new case that slipped past a stale enum list? On failure, don't crash and don't silently accept the bad value — send a corrective follow-up turn that names the specific problem ("the line items summed to 42.50 but total was given as 45.00 — re-check the total"), append the original tool_use block plus a tool_result with is_error: true carrying that message, and let the model retry with the concrete correction in front of it. A generic "that was wrong, try again" retry wastes a turn: the model has no more information the second time than the first.

Always cap the retry count — two or three attempts, then fail loudly into a human-review path. An unbounded retry loop against a task the model genuinely can't do isn't more robust, it's just a slower, more expensive failure that still ends in an error, now with extra latency and token spend attached.

Common exam distractor

Asking Claude to "return valid JSON" in a plain-text prompt, without a tool schema, and then parsing the response text directly, is a fragile pattern the exam likes to present as a wrong answer — free-text JSON is more prone to formatting drift than a forced tool call. So is treating tool_choice: "auto" as sufficient when the requirement is guaranteed structured output: "auto" still lets the model return plain text instead.

Key concept

tool_use with a schema eliminates syntax errors, not semantic ones. Nullable/optional fields are the schema-level defence against fabrication; a bounded validation-and-retry loop in your own code is the defence against everything a schema structurally cannot check.

Exam traps

Requesting JSON via prompt instruction alone and parsing the raw text response, instead of using a tool schema.

Free-text JSON generation is more prone to formatting errors (trailing commas, prose before/after the JSON) than a forced tool call, which commits the model to a defined schema and hands back an already-parsed object.
Using tool_choice "auto" when the requirement is guaranteed structured output.

"auto" lets the model return plain text instead of calling a tool - no structural guarantee at all. "any" guarantees a tool call while leaving tool selection flexible; a named tool forces one specific call.
Making every field in an extraction schema required to maximise data completeness.

Required fields pressure the model into fabricating a plausible-looking value when the source document genuinely lacks that information. Nullable/optional fields let the model return null honestly instead.
Believing a schema-conformant tool call means the extracted data is correct, or retrying validation failures without a cap.

A schema guarantees shape, not correctness - sum mismatches, misplaced values, and fabricated-but-well-typed fields all still need a separate validation step; and retrying that validation forever on a task the model can't do is just a slower failure, not a fix.

Practice question

An extraction feature asks Claude to "respond with a JSON object containing name and age" in plain text, then calls a JSON parser directly on the response. It occasionally fails because the response includes a sentence of preamble before the JSON. What's the more robust design?

A Add a regex to strip text before the first { character.
This is a brittle patch for a fragile pattern - it doesn't fix the underlying reliability gap and can still break on other formatting variations.
B Define name and age as a tool's input_schema and force that tool call with tool_choice, then read the fields from the tool_use block directly. Correct
This is the reliable structured-output pattern - the tool_use block's input is already a parsed, schema-conformant object, with no free-text parsing needed.
C Lower max_tokens so there's less room for preamble text.
A shorter cap doesn't reliably prevent preamble and risks truncating the JSON itself.
D Ask the model twice and use whichever response parses successfully.
This doesn't fix the root cause and adds cost/latency for a problem the tool-schema approach solves directly.

Build exercise: Build a validate-and-retry extraction loop

Intermediate · 40 minutes

You'll practice:

Force structured output via a tool schema and the correct tool_choice mode
Design nullable/enum fields that prevent fabrication
Add a bounded validation/retry loop for semantically invalid output

Define a tool with a schema that includes a constrained enum field (e.g. a status field limited to "paid", "unpaid", "overdue", "unclear") plus at least one nullable field, and force that specific tool call with tool_choice.

This sets up the schema-design half of the lesson - nullable fields and an escape-valve enum value are what actually prevent fabrication, before validation code even runs.

You should see: A schema-conformant tool_use response every time, with the nullable field returning null (not an invented value) on a test document that doesn't mention it.

Hints

Which field values would a real invoice sometimes just not contain? Those are your nullable-field candidates.

Use JSON Schema's type: ["string", "null"] for optional fields, and add an "unclear" option to any enum where the source could genuinely be ambiguous.

const tool = {
  name: "extract_invoice",
  description: "Extract structured invoice data",
  input_schema: {
    type: "object",
    properties: {
      invoice_number: { type: "string" },
      due_date: { type: ["string", "null"] },
      status: { type: "string", enum: ["paid", "unpaid", "overdue", "unclear"] },
    },
    required: ["invoice_number", "status"],
  },
};
const res = await client.messages.create({
  model: "claude-sonnet-5", max_tokens: 512,
  tool_choice: { type: "tool", name: "extract_invoice" },
  tools: [tool],
  messages: [{ role: "user", content: documentText }],
});

Add a second tool for a different document type (e.g. extract_receipt) and call the API twice: once with tool_choice "auto" and once with tool_choice "any", using an ambiguous input. Compare stop_reason across the two calls.

This is the exam's core tool_choice distinction made concrete - "auto" can legitimately skip the tool entirely, while "any" cannot, even when the model is unsure which tool fits.

You should see: With "auto", at least one run where stop_reason is end_turn and the model returned text instead of a tool call. With "any", stop_reason is tool_use every time, with the model picking whichever tool it judges fits best.
Hints
1. ```
What does the response's stop_reason field tell you about whether a tool was actually called?
```
2. ```
response.stop_reason will be "tool_use" when a tool was called and "end_turn" when the model replied with plain text - that's the signal to compare across the two tool_choice settings.
```
3. ```
for (const choice of [{ type: "auto" }, { type: "any" }]) {
  const res = await client.messages.create({
    model: "claude-sonnet-5", max_tokens: 512,
    tool_choice: choice,
    tools: [extractInvoiceTool, extractReceiptTool],
    messages: [{ role: "user", content: ambiguousText }],
  });
  console.log(choice.type, res.stop_reason);
}
```
Write a validation function in your own code that checks a semantic rule the schema can't enforce (e.g. that a line-items array sums to the stated total), run it against the tool call's input, and treat a mismatch as a validation failure.

This tests the exact gap a schema alone doesn't close - a well-formed, fully schema-conformant response that is still numerically wrong.

You should see: A schema-conformant response most of the time, and your validation code catching the case where the total field doesn't match the sum of the line items, even though every field individually satisfies the schema.
Hints
1. ```
The schema already guarantees each field's type - what check needs actual arithmetic, not type-checking?
```
2. ```
Sum the line_items amounts in your own code and compare to the total field with a small tolerance for floating point; treat a mismatch as a validation failure object, not a thrown exception you can't recover from.
```
3. ```
function validate(input) {
  const sum = input.line_items.reduce((s, li) => s + li.amount, 0);
  if (Math.abs(sum - input.total) > 0.01) {
    return { valid: false, reason: `line items sum to ${sum} but total was given as ${input.total}` };
  }
  return { valid: true };
}
```

On a validation failure, send a corrective follow-up turn: append the original assistant tool_use block, then a user message containing a tool_result with is_error true and the specific validation reason, and re-call the API.

A specific corrective message gives the model new information to act on; a generic "try again" retry does not, and typically reproduces the same mistake.

You should see: The retried call receives the corrective tool_result, and the second response either fixes the total or produces a materially different, self-consistent answer - not a repeat of the exact same wrong total.

Hints

What does the model need to see in the next turn to know specifically what to fix, rather than just that something was wrong?

Push the assistant's tool_use content block onto messages as the assistant turn, then push a user message with a single tool_result content block referencing that tool_use_id, is_error: true, and your validation reason string as content.

messages.push({ role: "assistant", content: res.content });
messages.push({
  role: "user",
  content: [{
    type: "tool_result",
    tool_use_id: toolUseBlock.id,
    is_error: true,
    content: `Validation failed: ${validation.reason}. Re-check the total against the line items.`,
  }],
});
const retry = await client.messages.create({ model: "claude-sonnet-5", max_tokens: 512, tool_choice: { type: "tool", name: "extract_invoice" }, tools: [extractInvoiceTool], messages });

Cap the validate-retry loop at 3 attempts total, and route to a human-review path (rather than crashing or looping forever) if validation still fails on the final attempt.

This is the safety-net half of the pattern - a validation loop without a cap turns a genuine extraction failure into an expensive, indefinite retry instead of a bounded, recoverable one.

You should see: A loop that retries up to 3 times, and on a deliberately unfixable test case (e.g. a document with an internally inconsistent total that no re-read will resolve), exits after attempt 3 into a clearly logged human-review branch rather than looping indefinitely.
Hints
1. ```
Where does the attempt counter live relative to the retry loop, and what should happen when it's exhausted?
```
2. ```
Wrap the call-validate-correct cycle in a for loop bounded at 3, and on the final failed attempt, break out and push the case onto a review queue instead of retrying again.
```
3. ```
const MAX_ATTEMPTS = 3;
let result;
for (let attempt = 1; attempt <= MAX_ATTEMPTS; attempt++) {
  result = await extractAndValidate(messages);
  if (result.valid) break;
  if (attempt === MAX_ATTEMPTS) {
    await routeToHumanReview(result);
    break;
  }
  messages = appendCorrection(messages, result);
}
```

Sources

Reused/adapted from CCAR-F lessons 4-3-structured-output and 4-4-validation-retry-loops