PPassTheCertExam

Study guides / CCAR-P / Domain 1

Integration · Lesson 3 of 8

1.3 - Accuracy-Latency Trade-offs and Defending the Configuration

Weigh model tier, effort and thinking, retrieval depth and verification passes against latency, and justify a configuration with a measured frontier and a decision record rather than an assumption.

"Accuracy versus latency" is not one dial. It is a set of separate levers, each with its own curve, and an architect is expected to pick a point, defend it with evidence, and say what would make them change it. The scenario question rarely asks "which is best"; it asks which lever addresses the stated constraint, or which justification would actually persuade a reviewer.

LeverWhat it tends to buyWhat it tends to cost
Model tierCapability on hard tasksLatency and price; smaller tiers are faster and cheaper
Effort (output_config.effort)Thoroughness, especially on agentic workTokens and time; lower effort means less thinking and fewer, terser tool calls
ThinkingMulti-step reasoning qualityThinking tokens are billed as output and count toward max_tokens
Retrieval depth (top-k, reranking)Recall of the right evidenceMore input tokens, and one more stage if you rerank
Verification pass (second call or judge)Catches errors before users see themA whole extra round trip
Parallel subagentsWall-clock time on breadth tasksMuch higher total tokens: Anthropic reports agents use roughly 4 times and multi-agent systems roughly 15 times the tokens of a chat
StreamingPerceived latency (time to first token)Nothing on total time; more client complexity

Effort and thinking: what the docs actually say

Effort is set per request as output_config: {effort: ...} with levels low, medium, high, xhigh and max (which levels exist depends on the model). high is the default and is identical to omitting the parameter. Effort affects all output tokens, including tool-call arguments and thinking, so lower effort also means fewer and terser tool calls. Anthropic calls it a behavioural signal, not a strict token budget: at low effort Claude still thinks on hard problems, just less.

Four practical consequences:

Anthropic's latency guide adds a discipline point: first build a prompt that works well without constraints, then apply latency reductions, or you never learn what top performance looks like. Its other levers are a smaller, faster tier for speed-critical work (check the models page for current options), shorter prompts and outputs, and streaming. max_tokens is called a blunt instrument: hitting it truncates the answer, so it protects budgets but is not a way to make a normal response arrive sooner.

Latency means several things

Pin down which latency the requirement is about. Time to first token governs how responsive a chat feels, and streaming improves it. Total time governs a pipeline step or a tool-using agent, and streaming does not shorten it. Percentiles matter more than means: a p95 target is missed by exactly the long-tail requests that verification passes and retries add to. For work no one is waiting on, the trade reverses: the Batch API accepts higher latency for lower cost, and the same logic applies to anything you can move off the interactive path.

Two design patterns manage the trade rather than pick a side. An advisor setup (a beta tool) lets a faster executor model consult a stronger advisor mid-generation, so most tokens are produced at the cheaper model's rate. Escalation on failure runs the cheap configuration first and re-runs only the cases a checker rejects at a higher setting; it needs a cheap, reliable failure signal such as a validator or tests, and the failed cases pay for two attempts.

Key concept: defend with a frontier, not an anecdote

A configuration is justified when it meets the latency and quality requirement and no measured alternative achieves the same quality faster or cheaper. That is an argument from a score-versus-cost-and-latency curve on your own workload, not from a vendor benchmark or one impressive example.

Justifying a configuration with evidence

  1. State the requirement in measurable terms: latency percentile (or time to first token), a quality metric with a threshold, and a cost ceiling.
  2. Build an eval set from production-like tasks with automated outcome checks, including hard cases; the tier-by-effort curve differs by workload shape. Anthropic's guidance notes that lowering effort tends to cost little on research-style tasks but more on long agentic coding.
  3. Sweep one variable at a time (tier, effort, retrieval depth, verification on or off) with everything else, including the prompt, held identical.
  4. Record distributions (p50 and p95 latency), output tokens and cost per completed task, not per token. Anthropic's cost guidance makes that same point and says a multi-model design must beat the single-model curve across its range.
  5. Pick from the frontier, write the decision record (chosen point, rejected alternatives, evidence), then shadow-test on a slice of real traffic before cutover.
  6. Name the re-evaluation triggers: new model release, traffic mix change, a metric regression.

Retrieval depth follows the same logic. In Anthropic's contextual retrieval experiments, passing 20 chunks outperformed 5 or 10, and a reranker narrowed a wider candidate set (150) to the final 20. More chunks cost input tokens and latency, so treat top-k as another swept variable (see Lessons 1.5 and 1.6).

Common exam distractor

Beware the answers that sound decisive but skip the measurement: "use the largest model at maximum effort to be safe", "drop to the smallest model to hit the latency target", "lower max_tokens to speed things up", "add an LLM verification step to every request as insurance", or "reuse the effort setting from the previous model". The correct answer usually runs a controlled comparison against the stated SLO, then gates expensive steps so only the requests that need them pay.

Exam traps

Practice question

An insurance-claims assistant must return an answer within a p95 target of 4 seconds and reach at least 94% on the team's claim-triage eval. It currently uses the top model tier at maximum effort and makes a second LLM call to verify every answer: accuracy is 96% but p95 is 11 seconds. The team has not tried other settings. What should the architect do first?

  • A Switch to the smallest available model tier immediately, keep the verification call on every answer, and re-run the eval afterwards to check whether accuracy still clears the 94% threshold and the four-second p95 target

    It jumps to a single lever without testing the others. The smallest tier may fall below 94%, and keeping verification on every answer keeps the extra round trip that is inflating p95. A mid-tier or lower-effort configuration may meet both requirements.

  • B Lower max_tokens until responses arrive within four seconds, and add a prompt instruction to give the shortest complete answer so that truncation stays rare while p95 falls

    max_tokens truncates responses; it does not make the model faster for normal-length answers and would risk cut-off output. It is not a latency strategy.

  • C Run a controlled sweep on the eval set across model tier and effort, gate verification behind a deterministic check, choose the cheapest configuration that meets both accuracy and p95, and record the evidence Correct

    It tests the levers separately, uses the stated requirements as the pass condition, gates the expensive verification step, and produces a decision record. It is the only option that justifies the configuration rather than assuming it.

  • D Remove the verification call and keep the top tier at maximum effort, then re-measure p95 to confirm the second call was the main source of the extra seconds

    Verification may be worth keeping for the cases that need it, so removing it wholesale changes only one lever and discards a check that protects the 94% accuracy. The top tier at maximum effort may be the larger part of the latency, and it is left unexplored.

Build exercise: Sweep tier and effort and write a defensible configuration record

Intermediate · 90 minutes

You'll practice:

  1. Write a one-page requirement sheet for a workload you know: latency target (percentile or time to first token), quality metric and threshold, cost ceiling, and whether the user is waiting.

    Without a stated requirement there is nothing to justify a configuration against. Whether a user is waiting decides if latency or cost dominates.

    You should see: A sheet with four to six lines, each measurable, such as 'p95 end-to-end under 4 s' and 'at least 94% on the triage eval'.

    Hints 
    1. Which of your requirements is a hard constraint and which is a preference you would trade?
    2. Write each requirement as a number with a unit and a way to measure it. Mark hard versus soft.
    3. Example: p95 latency <= 4 s (hard). Accuracy >= 94% on eval v3 (hard). Cost per completed task under the current baseline (soft). User is waiting: yes, so streaming is required.

  2. Build an eval set of at least 30 tasks with automated pass/fail checks, labelled by difficulty (easy, typical, hard).

    The accuracy-versus-cost curve differs by workload and difficulty, so the set must include the hard cases that low effort tends to fail.

    You should see: A file of tasks with an expected outcome or a checker function each, and a difficulty label.

    Hints 
    1. What would a checker look like that needs no human and no second LLM?
    2. Prefer deterministic checks (schema validity, exact fields, unit tests). Only add an LLM judge for criteria you cannot check deterministically.
    3. Structure: [{'id': 1, 'input': '...', 'difficulty': 'hard', 'check': 'fields_match'}]. Keep 10 of each difficulty and freeze the set before the sweep.

  3. Write the sweep harness. For each (model, effort) pair run every task, and record wall-clock seconds, output tokens and pass/fail.

    The harness makes the comparison repeatable. Everything except the swept variable must stay identical, including the prompt.

    You should see: A table with one row per (model, effort, task) and columns for seconds, output_tokens and pass.

    Hints 
    1. How will you keep the prompt byte-identical across runs?
    2. Load the prompt once, vary only model and output_config, and run each pair at least three times to see the spread.
    3. import time, anthropic
client = anthropic.Anthropic()
def run(model, effort, prompt):
    t0 = time.perf_counter()
    r = client.messages.create(model=model, max_tokens=4096, output_config={'effort': effort}, messages=[{'role': 'user', 'content': prompt}])
    text = next(b.text for b in r.content if b.type == 'text')
    return {'secs': time.perf_counter() - t0, 'out_tokens': r.usage.output_tokens, 'text': text}
Use model IDs from the current models page, and only effort levels your model supports.

  4. Add gated verification: run a cheap deterministic check on each answer and call a second model pass only when the check fails. Compare against always-verify and never-verify.

    This measures what verification actually buys and what it costs in latency, and shows why gating beats running it on every request.

    You should see: Three rows for the chosen configuration: never verify, always verify, verify on failure, each with accuracy and p95.

    Hints 
    1. On which fraction of requests does the gate fire, and what do those requests cost in extra latency?
    2. Record the gate rate. Verify-on-failure adds latency only to the failing fraction, but if that fraction exceeds 5% of traffic those slow requests sit inside your p95, so check the percentile rather than assuming the gate is free.
    3. Pseudocode: ans = run(...); if not check(ans): ans = verify(ans). Report p95 over all requests, not just verified ones, because the tail is what the requirement measures.

  5. Choose a configuration from the results and write the decision record: chosen point, its measured accuracy, p50/p95 and cost per completed task, two rejected alternatives with reasons, and the triggers that would reopen the decision.

    This is the defence a reviewer will read. A configuration without rejected alternatives and evidence is an opinion.

    You should see: A half-page record that a reviewer could verify by re-running your harness.

    Hints 
    1. If a sceptic asked 'why not the next cheaper setting?', where in your data is the answer?
    2. Select the cheapest row that satisfies every hard requirement. For each rejected row name the requirement it failed or the cheaper row that dominated it.
    3. Example: Chosen: mid tier, medium effort, verify on failure: 95% acc, p95 3.4 s. Rejected: low effort (91%, fails accuracy). Rejected: top tier max effort (96% but p95 9 s). Reopen if a new model ships, eval accuracy drifts by more than 2 points, or traffic shifts to longer claims.

Sources