PPassTheCertExam

Study guides / CCAR-P / Domain 3

Evaluation, Testing & Optimisation · Lesson 5 of 6

3.5 - Optimising Token Usage, Latency and Cost-Performance

Measure where tokens, time and money go, then match each cost or latency driver to the right lever (caching, batching, tiering, effort, output shaping, context trimming) and validate every trade-off against your evals.

Optimisation questions at the Professional level are rarely “name a technique”. They are “here is a workload and a measurement, which lever fits, and what does it cost you in quality?” The method has three rules. Measure first, so you know which driver dominates. Take the free wins first: caching, batching and input hygiene lower cost or latency without lowering output quality. Validate every trade-off: effort, model tier and multi-model designs exchange cost for capability, so each is adopted only after your evaluation set (Lessons 3.1 and 3.2) shows quality holds. The unit to optimise is cost per completed task, not cost per token: a cheaper configuration that fails or retries more often is not cheaper.

Anthropic’s latency guidance adds a sequencing caution: first get a prompt working well, then reduce latency, because doing it prematurely can hide what top performance looks like.

Measure before you optimise

Every response reports token usage: uncached input, cache-creation input, cache-read input and output. Total input is the sum of the first three. Aggregate these by workload step and by model, and record latency as both time to first token and total time. Three tools help:

Match the driver to the lever

Dominant driverLeverWhat it costs you
Large stable prefix (system prompt, tools, documents) resent on every callPrompt cachingWrite premium; any prefix change is a miss; cache management
Work nobody waits for (nightly jobs, offline evals)Message Batches APIAsynchronous, up to 24 hours, unordered results
Easy steps running on a strong tierTiering or routingRouter errors; more moving parts; needs evals
Thinking and tool-call depth dominateLower effort first, then a smaller tierCapability reduction, so validate
Verbose outputOutput shapingBlunt limits truncate; wording can hurt quality
Bulky tool results or history accumulatingRetrieval instead of inlining, context editing, trimmingCan invalidate cache; risk of dropping needed context
User-perceived waitStreamingPerceived latency only

Prompt caching: what an architect needs to know

Caching reuses a processed prefix. Requests are assembled in the order tools, then system, then messages, and a change at one level invalidates that level and everything after it. Cache hits need a 100% identical prefix up to and including the block carrying the breakpoint. Put stable content first and the breakpoint after it. You can set breakpoints explicitly (up to four per request) or use automatic caching, which places the breakpoint on the last cacheable block and moves it forward as a conversation grows. Lookback from a breakpoint is limited to 20 blocks, so a fast-growing conversation may need another breakpoint.

The default lifetime is five minutes, refreshed on use, with a one-hour option. At the time of writing the docs price a five-minute write at 1.25 times the base input price, a one-hour write at 2 times, and a cache read at 0.1 times (lower on some models); confirm on the pricing page. The economics: a prefix read even once within its lifetime costs less in total than sending it uncached twice (1.25 + 0.1 against 2), so caching pays on repetition and loses on one-off content. Minimum cacheable length varies by model, so a short prefix may not cache at all.

Things that break the cache are architectural, not incidental: editing tool definitions or their order, toggling web search or citations, changing tool_choice, adding or removing images, changing thinking parameters, or changing the effort setting between requests. Hold these constant inside a cached session (a beta per-message effort change on some models preserves the cache). Track the cache-read share of input tokens as a first-class metric. And remember what caching does not do: it reduces the cost and latency of reprocessing input, and does nothing for how fast output tokens are generated.

Common exam distractor

“Caching speeds up generation” is wrong: it only removes reprocessing of a repeated input prefix. Equally tempting and wrong: downgrading the model first, treating caching, batching and routing as alternatives (they stack), using batches for an interactive path, and assuming batch results arrive in submission order (match on custom_id).

Batching, tiering, effort and multi-model designs

Batches. The Message Batches API bills all usage at 50% of standard prices, with most batches finishing within an hour and a hard expiry at 24 hours. A batch is limited to 100,000 requests or 256 MB, results may come back in any order, and streaming is not supported. Caching inside batches is best-effort because requests run concurrently, so use the one-hour lifetime for shared prefixes. It suits offline evals (Lesson 3.3), backfills and nightly jobs, not user-facing requests.

Tiering and effort. Anthropic describes two starting points: efficiency-first (start on the fast, low-cost tier and upgrade only for demonstrated capability gaps) and capability-first (start on a strong model, then lower effort or change tier once evals justify it). It also states that tuning effort is often a better lever than switching models. Set effort explicitly, sweep it against your evals, and re-sweep when you change models. Routing (a cheap classifier sends only hard requests to a stronger tier) keeps expensive spend proportional to difficulty, but the router has its own error rate and cost.

Multi-model designs. Anthropic’s guidance describes two patterns: an advisor (a cheaper executor consults a stronger model at hard decisions) and an orchestrator (a stronger model delegates bulk independent work to cheaper workers). They pay off only in specific shapes: a real capability gap with a reliable consultation signal, or work that genuinely splits into independent pieces or exceeds one context window. For one dependent chain that fits in a context, a single model at lower effort is usually cheaper. The stated measurement order is: sweep effort on the current model, price the stronger model alone at low effort, and treat that number as the baseline any multi-model design must beat.

Output shaping and context trimming

Ask for concision directly, limiting by sentences or paragraphs rather than word counts, which models follow less reliably. max_tokens is a hard cap that can cut an answer mid-sentence, so it suits short answers and safeguards, not length shaping. On some current models effort controls thinking volume rather than visible length, so prompt for length separately. To trim context, move large reference material behind retrieval, prune unused tool definitions, and consider context editing, which clears old tool results as a conversation grows; clearing invalidates cached prefixes, so clear enough at a time to justify each cache rewrite. Context budgeting is covered in Lesson 6.4 and caching design in Lesson 6.5.

Key concept: an optimisation is a claim you must test

Each change should state its driver, its expected saving and its quality risk, and be accepted only when cost per completed task drops while the gate metrics from Lesson 3.1 hold. If the saving is real but quality regressed, you have not optimised; you have moved the cost somewhere else.

Exam traps

Practice question

A team runs three workloads: an interactive support chat with an 8,000-token stable system prompt and tool set, where users typically send several messages within a few minutes; a nightly job classifying tens of thousands of documents; and a triage step where most requests are routine. Finance wants a large cost reduction. The engineering lead proposes moving everything to the smallest model tier. What should the architect do first?

  • A Approve the move, since the smallest tier has the lowest per-token price and will lower cost for all three workloads without any change to the architecture or extra engineering effort, and the saving can be reported next week.

    Per-token price is not cost per completed task. Failures and retries could raise total cost, and the change gives up capability before any free win has been applied or validated.

  • B Measure cost and latency drivers per workload from usage data, then apply matched levers: caching for the chat prefix, batching for the nightly job, evaluated tiering or effort for triage, each accepted only if eval gates hold. Correct

    Each lever addresses a specific driver. Free wins come first, trade-offs are validated on an eval set, and the outcome is judged as cost per completed task.

  • C Move all three workloads to the Batches API to get the 50% discount everywhere, since a flat discount is the largest single saving available, keeps operations simple and needs no per-workload analysis.

    Batching suits only the nightly job. The interactive chat needs immediate responses, and batch requests are asynchronous with a window of up to 24 hours.

  • D Enable prompt caching everywhere and stop, because caching is the single lever that reduces cost for every workload, whatever its traffic pattern and latency need, and it needs no evaluation to adopt.

    Caching helps repeated prefixes, but it does nothing for the nightly job's batch economics or triage difficulty routing, and short one-off prompts may not benefit at all.

Build exercise: Profile, optimise and defend: caching, batching and an evaluated tier or effort change

Intermediate · 80 minutes

You'll practice:

  1. Choose a real or realistic application with at least three steps or workloads. For each, log or estimate input tokens, output tokens, call volume, latency need (interactive or not) and how much of the input is stable across calls. Use the token counting endpoint to measure the stable prefix. Name the dominant driver per workload.

    You cannot pick a lever until you know the driver. The token counter lets you size prefixes before spending anything.

    You should see: A table with one row per workload: tokens in and out, volume, latency need, stable-prefix size and a one-line driver diagnosis.

    Hints 
    1. Which of your workloads resend the same content on every call, and which are one-off?
    2. Count only the stable content (system prompt plus tools) with the token counting call, then compare it to the per-call variable input.
    3. import anthropic
client = anthropic.Anthropic()
MODEL = "claude-sonnet-5"   # check the models page; count against the model you will run

stable = client.messages.count_tokens(
    model=MODEL, system=SYSTEM_PROMPT, tools=TOOLS,
    messages=[{"role": "user", "content": "x"}],
).input_tokens
print("stable prefix tokens (approx):", stable)

  2. For the interactive workload, add a cache breakpoint after the stable system content, send two requests within the cache lifetime, and read the usage fields. Then change one character in the system prompt and send again. Finally, change the effort setting between two otherwise identical requests and observe the cache fields.

    Seeing the write, the read, the exact-prefix miss and the effort-induced miss with your own numbers is far more durable than memorising the rules.

    You should see: First call: cache_creation_input_tokens above zero. Second call: cache_read_input_tokens above zero. After the edit or the effort change: a new cache write and a read of zero for the changed section.

    Hints 
    1. Where in the request does the breakpoint go so that it covers everything stable but nothing that varies per call?
    2. Put cache_control on the last stable content block (the system block or the last tool), keep the user message after it, and print the three input-token fields from usage each time.
    3. system = [{"type": "text", "text": SYSTEM_PROMPT, "cache_control": {"type": "ephemeral"}}]
def ask(q, **extra):
    r = client.messages.create(model=MODEL, max_tokens=300, system=system,
                               messages=[{"role": "user", "content": q}], **extra)
    u = r.usage
    print("uncached:", u.input_tokens, "write:", u.cache_creation_input_tokens, "read:", u.cache_read_input_tokens)
ask("First question")
ask("Second question")                                    # expect a cache read
ask("Third question", output_config={"effort": "low"})   # effort change: expect a fresh write

  3. Run the nightly workload as a batch: one request per document with a meaningful custom_id, poll until processing ends, and read results by custom_id. Deliberately add one invalid request and confirm how it appears in the results. If you have documents that share a long prefix, use the one-hour cache lifetime on it.

    Batching is the free win for latency-insensitive work, and correct result handling (unordered, per-request errors) is what the exam tests.

    You should see: A completed batch whose results are matched by custom_id, with counts of succeeded and errored requests and the invalid one reported as errored rather than crashing the run.

    Hints 
    1. If results come back in a different order from submission, what field lets you re-associate each with its input?
    2. Create the batch, retrieve until processing_status is ended, then iterate over the results and branch on the result type.
    3. import time
batch = client.messages.batches.create(requests=[
    {"custom_id": f"doc-{i}", "params": {"model": MODEL, "max_tokens": 200,
      "messages": [{"role": "user", "content": f"Classify this document:\n\n{text}"}]}}
    for i, text in enumerate(documents)])
while client.messages.batches.retrieve(batch.id).processing_status != "ended":
    time.sleep(30)
for item in client.messages.batches.results(batch.id):
    if item.result.type == "succeeded":
        print(item.custom_id, next(b.text for b in item.result.message.content if b.type == 'text')[:60])
    else:
        print(item.custom_id, "->", item.result.type)

  4. For the triage or reasoning-heavy workload, run your evaluation set across three configurations: current, lower effort on the same model, and a smaller tier. Confirm model IDs and effort support on the current models page. Record pass rate on the gate metrics, latency and cost per completed task for each, and reject any configuration that breaks a gate.

    This is the validated trade-off step. Effort is the cheapest experiment, and the result tells you whether a routing or multi-model design is even worth building.

    You should see: A comparison table where each configuration has gate pass or fail, p95 latency and cost per completed task, and at least one configuration explicitly rejected with the reason.

    Hints 
    1. If a cheaper configuration passes accuracy but needs more retries, what happens to cost per completed task?
    2. Score every configuration on the same fixed eval set, count retries against the task, and compute cost from usage using rates you looked up on the pricing page.
    3. configs = [("claude-sonnet-5", "high"), ("claude-sonnet-5", "low")]   # add a smaller tier if it supports your needs
for model, effort in configs:
    rows = run_eval(model=model, effort=effort)            # your fixed eval set, returns per-task results
    print(model, effort,
          "gate pass:", all(g(rows) for g in GATES),
          "cost/task:", round(sum(r["cost"] for r in rows) / len(rows), 5),
          "p95 s:", pct([r["seconds"] for r in rows], 95))

  5. Write a decision record that lists each optimisation you adopted or rejected, the driver it addressed, the measured cost per completed task before and after, the quality gates checked, and the monitoring you will add (cache-read share, batch share, per-tier cost) so a regression is visible.

    The exam wants justified trade-offs, and monitoring keeps optimisations from silently decaying when prompts, tools or models change.

    You should see: A one-page record with at least one adopted and one rejected optimisation, each tied to a measurement and a gate, and a named owner for the monitoring.

    Hints 
    1. Which of your optimisations would silently stop working if someone reordered the tool definitions?
    2. For each entry, write driver, change, evidence, risk, and the metric that would show it failing. Include cache-read share for anything relying on caching.
    3. Example entry: Driver: 8k-token stable prefix resent on every chat turn. Change: cache breakpoint after tools and system, tool order frozen in code. Evidence: cache-read share of input tokens measured after warm-up, cost per resolved chat before and after from usage data. Gate: resolution rate unchanged on the eval set. Risk: any tool or prompt edit resets the cache. Monitor: alert if cache-read share falls below the level measured in the pilot.

Sources