Model Selection & Optimisation · Lesson 5 of 5

2.5 - Cost Optimisation Techniques

Combine caching, batching, input hygiene, and model routing into one coherent cost strategy - free wins first, capability trade-offs last.

Cost optimisation techniques split into two categories, and the order you reach for them matters. Free wins — prompt caching, the Batches API, input-token hygiene, and agent-loop hygiene — lower what you pay without lowering output quality, so they're applied first and left on permanently. Trade-offs — effort/task budgets (Lesson 2.4), model tier (Lesson 2.1), and multi-model architectures — exchange cost for capability, so they're reached for only after the free wins are exhausted, and validated against an evaluation set every time. The exam consistently rewards this ordering: an answer that reaches for a model downgrade before caching, batching, or input hygiene are in place is giving up capability it didn't need to give up yet.

Free wins: caching, batching, input hygiene

Prompt caching (Lesson 1.4/2.2) is typically the single largest lever in an agentic loop — because every turn resends the growing conversation history, an uncached loop's cost grows roughly with the square of turn count, and caching reprices everything already seen down to about a tenth of its input cost. The Batches API (Lesson 1.7) discounts every token type — input, cache reads, cache writes, and output — by 50%, but only applies to latency-insensitive, asynchronous work like a nightly summarisation job; it cannot run a mid-batch tool loop, so each batched request is single-shot. Input hygiene means sending the model only what a task needs and letting it fetch the rest: moving a large reference document behind a retrieval tool instead of inlining it in every prompt, pruning tool schemas that most requests don't use, and downscaling images to the resolution the task actually requires rather than sending everything at full size.

Trade-offs: effort, budgets, and model routing

Once the free wins are in place, the remaining levers change what the model can do, so they're applied last and always measured. Effort and task budgets (Lesson 2.4) are the first trade-off to reach for — cheaper to tune and validate than a model swap. Model routing — sending each step to the cheapest tier that meets its quality bar — cuts cost by matching spend to actual difficulty rather than raw request volume (Lesson 2.3's stepping-down method). Beyond a single model, two-model architectures can help in specific, narrow shapes: an advisor pattern (a cheaper model runs the loop and consults a stronger one only on genuinely hard decisions) pays off when the capability gap is wide and the cheap model reliably recognises when to escalate — a fragile condition, since a weak escalation signal can make the pairing worse than the strong model alone. An orchestrator pattern (a strong model plans and delegates bulk, independent sub-tasks to cheaper workers) pays off only when there's genuine bulk fan-out to hand off; for a single dependent chain of work, one model at well-tuned effort usually beats the overhead of planning, delegating, and merging.

Matching the cost driver to the lever

A useful exam habit is to name the specific cost driver before picking a technique, rather than reaching for a favourite lever out of habit. A system prompt and tool schema re-billed on every call points to caching, not a cheaper model. A large reference document inlined into every prompt points to moving it behind a retrieval tool, not batching. Work nobody is waiting on synchronously points to the Batches API, not effort tuning. Bulky tool results piling up across a long agentic loop point to context editing, compaction, or a client-side prune at natural boundaries — not a model downgrade, which wouldn't touch the accumulation at all. Thinking and tool-call depth dominating spend, with headroom in an eval, points to effort first and a tier drop only after that's exhausted. Picking the technique that actually matches the driver is what separates a coherent strategy from a grab-bag of unrelated changes.

A simple routing pattern

One common pattern: a fast, cheap model classifies or triages a request first; only requests that actually need it get escalated to a stronger, more expensive model. This keeps the expensive tier's spend proportional to genuine difficulty rather than to raw request volume — and it composes cleanly with caching (the triage model's own prompt can itself be cached) and with batching (if the triage step doesn't need to be synchronous).

Common exam distractor

An answer presenting caching, batching, and model routing as mutually exclusive alternatives — pick one — is a trap. In practice they stack, and a mature system typically applies more than one at once.

Key concept

Optimise cost per completed task, not cost per token or cost per request. A configuration that looks cheaper on paper but fails more often, or needs more retries or escalations to finish the job, isn't actually cheaper.

Exam traps

Treating prompt caching, batching, and model routing as competing options where only one should be chosen.

These techniques address different cost drivers (repeated input, latency-insensitive volume, and per-task difficulty respectively) and typically combine rather than substitute for each other.
Applying trade-off levers (lower effort, a cheaper model) before exhausting the free wins (caching, batching, input hygiene).

Free wins reduce cost without touching output quality and should be applied first. Trade-offs exchange cost for capability and should only be reached for afterward, so you aren't giving up quality you didn't need to give up.
Assuming a two-model advisor or orchestrator architecture is always cheaper than one well-tuned model.

A two-model setup beats one model only in specific shapes: a wide capability gap with a reliable consult-gating signal, or genuinely bulk, independent fan-out work. For a single dependent chain of work, one model at tuned effort often beats the planning/delegating/merging overhead of orchestration.
Judging a cost optimisation by cost per token rather than cost per completed task.

A cheaper-looking configuration that fails more often, or needs retries or human escalation to finish the job, can cost more overall once those downstream costs are counted. The correct unit of measurement is cost per completed task.

Practice question

A high-volume application has a large repeated system prompt, a nightly batch summarisation job, and a live triage step that only sometimes needs deep reasoning. Which combination of techniques fits best?

A Prompt caching for the repeated system prompt, the Batches API for the nightly job, and a cheap-tier-first routing pattern for triage. Correct
Each technique is matched to the specific cost driver it addresses - this is exactly the composed strategy the lesson describes.
B Prompt caching alone, applied everywhere, since it's the single most effective lever.
Caching addresses repeated-input cost specifically; it doesn't address the batch job's throughput needs or the triage step's difficulty-based routing opportunity.
C Switch every call in the system to the cheapest available model tier.
This ignores quality requirements on steps that need stronger reasoning, and doesn't address the batch job's latency-insensitive volume at all.
D Disable streaming across the system to reduce overhead.
Streaming affects delivery, not the underlying token cost - it isn't a cost-optimisation technique for this scenario.

Build exercise: Build and validate a layered cost strategy: caching, batching, and routing together

Intermediate · 40 minutes

You'll practice:

Combine caching, batching, and model routing into one coherent strategy
Implement each technique with real SDK calls
Measure combined savings in cost per completed task, not per token

If you have logged usage data, aggregate response.usage by application step (or, if not, sketch three hypothetical steps of varying difficulty and latency needs) to identify where token spend concentrates.

Designing a composed strategy from a real or realistic cost profile, not just naming techniques abstractly, is the actual exam-relevant skill.

You should see: A short table or printout: step name, total input/output tokens, call volume - enough to justify which technique applies where.

Hints

If you don't have real usage logs, what three steps would a typical Claude-backed application have, with clearly different volume and difficulty profiles?

Group logged usage records by a 'step' field and sum input/output tokens and call counts per group; if you have no logs, just write out three hypothetical steps (e.g. triage, nightly summarisation, escalated response) with rough relative volume and difficulty.

import json

with open("usage_log.jsonl") as f:
    records = [json.loads(line) for line in f]

by_step = {}
for r in records:
    by_step.setdefault(r["step"], []).append(r)

for step, recs in by_step.items():
    total_in = sum(r["input_tokens"] for r in recs)
    total_out = sum(r["output_tokens"] for r in recs)
    print(step, "input:", total_in, "output:", total_out, "calls:", len(recs))

Implement prompt caching on the step with a large, stable repeated system prompt, using a 1-hour TTL if calls to that step are more than five minutes apart.

This is the caching lever from Lesson 2.2 applied to a specific step, choosing a TTL based on actual call spacing rather than defaulting to the 5-minute TTL everywhere.

You should see: A system block with cache_control set, and (on a second call) a nonzero cache_read_input_tokens.

Hints

Besides adding cache_control, what else might you want to configure if calls to this step are infrequent?

Set cache_control to {type: 'ephemeral', ttl: '1h'} on the stable system content if calls are spaced more than five minutes apart - the longer TTL avoids repeated cache-write premiums on a slow-moving step.

system_prompt = [{
    "type": "text",
    "text": SUPPORT_POLICY_DOC,
    "cache_control": {"type": "ephemeral", "ttl": "1h"},
}]

response = client.messages.create(
    model="claude-sonnet-5", max_tokens=512,
    system=system_prompt,
    messages=[{"role": "user", "content": user_message}],
)

Implement the nightly summarisation job using the Batches API instead of synchronous calls, submitting one request per document and polling until the batch completes.

The nightly job is latency-insensitive by nature - this is exactly the workload shape the Batches API's 50% discount is meant for, and the exam expects you to recognise that shape.

You should see: A batch created with one request per document, a poll loop that waits for processing_status to reach 'ended', then results read by custom_id.

Hints

Batch results come back in what order, and how do you know which result belongs to which input document?

Give each request a custom_id tied to its document, poll batches.retrieve until processing_status is 'ended', then iterate batches.results and match on custom_id - never assume result order matches submission order.

batch = client.messages.batches.create(
    requests=[
        {
            "custom_id": f"summary-{doc_id}",
            "params": {
                "model": "claude-sonnet-5",
                "max_tokens": 1024,
                "messages": [{"role": "user", "content": f"Summarize:\n\n{doc_text}"}],
            },
        }
        for doc_id, doc_text in nightly_documents.items()
    ]
)

status = client.messages.batches.retrieve(batch.id)
while status.processing_status != "ended":
    time.sleep(30)
    status = client.messages.batches.retrieve(batch.id)

for result in client.messages.batches.results(batch.id):
    if result.result.type == "succeeded":
        print(result.custom_id, result.result.message.content)

Implement the triage step as a cheap-tier-first router: classify with Claude Haiku 4.5 first, and only escalate to Claude Opus 5 at high effort when the classification says the request is complex.

This is the routing pattern from the lesson, made concrete - most requests should resolve on the cheap tier, with the expensive tier's spend proportional to actual difficulty.

You should see: A function that returns a cheap-tier response for most inputs and only calls Opus 5 for inputs the triage step flags as complex.

Hints

What should the triage call's max_tokens be, given it only needs to output one word?

Keep the triage call cheap and short (a tiny max_tokens is enough for a one-word verdict), then branch on the verdict to decide which model handles the real response.

def triage(ticket_text):
    r = client.messages.create(
        model="claude-haiku-4-5", max_tokens=20,
        messages=[{"role": "user", "content": f"Is this ticket simple (routine, one-step) or complex (needs judgment)? Answer one word.\n\n{ticket_text}"}],
    )
    verdict = next(b.text for b in r.content if b.type == "text").strip().lower()
    if "complex" in verdict:
        return client.messages.create(
            model="claude-opus-5", max_tokens=1024,
            output_config={"effort": "high"},
            messages=[{"role": "user", "content": ticket_text}],
        )
    return client.messages.create(
        model="claude-haiku-4-5", max_tokens=512,
        messages=[{"role": "user", "content": ticket_text}],
    )

Compare total dollar cost of routing every triage request through Opus 5 (the naive baseline) against the layered routing strategy from step 4, using the real per-tier rates.

This closes the loop by measuring the combined strategy's actual savings rather than asserting it - the exam wants you to be able to justify a strategy with numbers, not just name the techniques.

You should see: A printed 'before' and 'after' dollar figure, with the routed strategy noticeably cheaper for a mixed batch of mostly-simple tickets.

Hints

You'll need usage objects from both the all-Opus-5 baseline and the routed strategy - how do you keep the rate lookup consistent across the two?

Collect usage objects from each configuration into separate lists, sum cost per list using each call's actual model rate, then compare the two totals and print the percentage saved.

def total_cost(usages, rates):
    return sum(
        u.input_tokens / 1e6 * rates[0] + u.output_tokens / 1e6 * rates[1]
        for u in usages
    )

before_cost = total_cost(baseline_usages, (5.00, 25.00))   # everything on Opus 5
after_cost = total_cost(routed_usages_haiku, (1.00, 5.00)) + total_cost(routed_usages_opus, (5.00, 25.00))
print(f"Before: ${before_cost:.2f}  After: ${after_cost:.2f}  Saved: {(1 - after_cost/before_cost)*100:.1f}%")

Sources

Anthropic docs - Prompt caching
Anthropic docs - Batch processing