Model Selection & Optimisation · Lesson 3 of 5

2.3 - Choosing a Model for a Task

Apply a concrete decision framework - task difficulty, latency budget, volume, cost sensitivity - and the stepping-down method, instead of picking a model by habit.

Four questions cover most of the decision: how hard is the reasoning this step actually requires; what's the acceptable latency; what's the call volume, since volume multiplies any per-call cost difference; and how much does an occasional wrong answer actually cost downstream. A high-volume, low-stakes, latency-sensitive step points toward a fast, cheap tier. A low-volume, high-stakes step where a wrong answer is expensive points toward the strongest tier, even at higher per-call cost. This framework is necessary but not sufficient on its own — the exam also expects you to know how to validate a candidate model, not just how to guess at one.

Unpacking the four questions

Each question does different work in the decision, and the exam tends to test whether you can isolate which one actually matters in a given scenario. Task difficulty asks whether the step genuinely requires multi-step reasoning, nuanced judgment, or careful handling of ambiguity — not whether the topic sounds hard; a legal-sounding request that's really a lookup-and-template-fill is easy, while a plain-language request that hides several conflicting constraints is hard. Latency budget asks how long a user or downstream system will tolerate waiting — an interactive chat turn has a tight budget; a nightly job has none, which is exactly the signal that also points toward the Batches API (Lesson 2.5). Volume asks how many times this step runs, because it's the multiplier that turns a small per-call price difference into a large total-spend difference — a $0.01 difference per call is irrelevant at 100 calls a month and decisive at 10 million. Cost of a wrong answer asks what happens downstream when the model is wrong — a mis-classified support ticket that a human re-routes in five seconds costs almost nothing; an autonomously executed wrong action with no review can cost real money, legal exposure, or user trust. A scenario question is often really asking you to identify which one of these four dominates, since they don't all point the same direction on every task.

Pricing the tail, not the median

On the typical, easy-difficulty case, every model tier tends to look similar — which makes the cheapest tier look like the obvious winner if that's all you measure. In practice, cost and error concentrate in the hardest slice of a workload: a handful of genuinely hard requests can carry a disproportionate share of both the spend and the risk, even when every other request is trivial. Comparing candidate models only on average-case prompts hides exactly the cases that determine whether a cheaper tier is actually safe to ship — always include some deliberately hard, edge-case prompts in any model comparison, not just representative ones.

The stepping-down method

Once a tier is provisionally chosen, the exam-correct way to look for savings is to move down the levers in order rather than jumping straight to a cheaper model: first sweep the effort parameter on the current tier (Lesson 2.4) against an evaluation set — if a lower effort level holds accuracy, that's a cheaper win with no model change at all. Only after effort is tuned do you consider dropping to the next tier down, confirm which parameters and effort levels that tier actually supports (they differ by tier — see Lesson 2.1), reset effort to that tier's own default rather than carrying over a hardcoded level, and re-sweep from there. One notch at a time, each validated against the same evaluation set, never several changes at once.

Re-evaluating over time

This isn't a one-time decision at launch. As a feature's volume grows, the cost side of the trade-off grows with it; as a cheaper tier's capability improves across model generations, a task that once needed the top tier may not any more — a task assigned to Opus 5 today might be handled just as well by Sonnet 5 once a future Sonnet generation closes the gap. Revisit the assignment against your eval set (Domain 8) periodically, not just once, and re-run the stepping-down sweep whenever the model generation, the prompt, or the workload shifts meaningfully.

Key concept

"What does a wrong answer cost here?" is often the single most useful question in this framework — it's what separates a nice-to-have quality bump from a genuine requirement for the strongest model.

Common exam distractor

An answer that jumps straight to the cheapest tier without first sweeping effort on the current tier, or that judges model fit from a single test run, is a trap. A real keep/revert decision needs the effort lever exhausted first and repeated trials against an eval — a difference of a task or two of pass rate on one comparison is noise, not a signal.

Exam traps

Treating a model-selection decision made at launch as permanent, without revisiting it as volume or model capability changes.

The cost side of the trade-off scales with volume, and relative model capability shifts over time - a choice that was right at launch may not stay right.
Jumping directly to the cheapest model tier without first sweeping effort/budget settings on the current tier.

Effort is a within-model lever that's cheaper and faster to tune and validate than a model swap. The stepping-down method sweeps effort before dropping tiers so the cost/quality trade is measured lever by lever, not conflated into one big untested jump.
Evaluating model choice only on typical, median-difficulty requests.

Cost and error concentrate in the hardest tail of a workload; a model that looks equivalent to a cheaper tier on typical cases can diverge sharply on the hardest slice - which is often where most of the spend and most of the downstream risk actually sits.
Deciding to keep or revert a model choice based on a single test run or one anecdote.

Differences of a task or two of pass rate are within normal noise on a single run. A defensible keep/revert decision needs repeated trials against a fixed evaluation set, not one comparison.

Practice question

A step in an application autonomously drafts and sends legal notices with no human review, at low volume. Which factor should weigh most heavily in model selection here?

A Latency, since users are waiting for the response.
The scenario describes an autonomous, low-volume, unreviewed step - latency isn't the dominant concern here.
B The cost of a wrong answer, since an incorrect autonomous legal notice with no human check is high-stakes despite low volume. Correct
Low volume keeps per-call cost differences small in absolute terms, while an unreviewed, high-stakes output makes the cost of an error the dominant factor - this points toward the strongest available tier.
C Call volume, since that's what drives total spend.
Volume matters for total cost, but this scenario is explicitly low-volume, so it isn't the deciding factor here.
D None of these - model choice has no bearing on output correctness.
Model capability does affect reasoning quality and error rate, which is directly relevant to a high-stakes, unreviewed task.

Build exercise: Build a small eval set and apply the stepping-down method

Intermediate · 35 minutes

You'll practice:

Apply a structured framework to a model-selection decision
Compare models against a fixed evaluation set including hard cases
Apply the stepping-down method: sweep effort before dropping tier
Measure cost per completed task including failures, and repeat trials to rule out noise

Assemble a small evaluation set for a real or hypothetical task: 8 easy cases and 2 deliberately hard edge cases, each with an expected answer.

Both 'price the tail' and the stepping-down method require comparing models on a fixed set that includes hard cases - without this you're guessing from vibes.

You should see: A list of 10 (prompt, expected answer) pairs, with at least 2 explicitly marked as hard.

Hints

What makes a test case 'hard' for a classifier - ambiguity, edge-case phrasing, or something else?

Write 8 straightforward classification prompts with a clear expected label, then 2 prompts that are ambiguous or combine multiple signals a simpler model might misread.

eval_cases = [
    {"prompt": "Classify: 'Where is my refund?' -> billing/technical/other", "expected": "billing", "hard": False},
    # ... 6 more easy cases ...
    {"prompt": "Classify: 'My subscription auto-renewed but the confirmation references a plan I cancelled two billing cycles ago, and support already told me it was cancelled' -> billing/technical/other", "expected": "billing", "hard": True},
]

Run the full eval set through Claude Haiku 4.5 and Claude Sonnet 5, and report accuracy separately for the easy cases and the hard cases on each model.

Splitting accuracy by difficulty is what 'price the tail' means in practice - a model can look fine on the blended average while failing specifically on the cases that matter most.

You should see: Similar (likely high) accuracy on easy cases across both models, with a visible gap on the hard cases.

Hints

How do you keep the easy-case and hard-case accuracy numbers separate rather than one blended score?

Run each case through the model, record whether the expected label appears in the response, tag each result with its 'hard' flag, then compute two separate accuracy figures.

def run_eval(model, cases):
    results = []
    for case in cases:
        r = client.messages.create(model=model, max_tokens=50, messages=[{"role": "user", "content": case["prompt"]}])
        text = next(b.text for b in r.content if b.type == "text").strip().lower()
        results.append({"correct": case["expected"] in text, "hard": case["hard"], "usage": r.usage})
    return results

for model in ["claude-haiku-4-5", "claude-sonnet-5"]:
    results = run_eval(model, eval_cases)
    easy = [r for r in results if not r["hard"]]
    hard = [r for r in results if r["hard"]]
    print(model, "easy:", sum(r["correct"] for r in easy)/len(easy), "hard:", sum(r["correct"] for r in hard)/len(hard))

Before considering a tier drop, sweep output_config.effort at low/medium/high on Claude Opus 5 against the same eval set, to see whether a cheaper effort setting already holds accuracy.

This is the first step of the stepping-down method - the exam expects effort to be tuned before a model tier change is even considered.

You should see: Accuracy that's flat or nearly flat across effort levels on this simple classification task, suggesting low effort is sufficient here.

Hints

What should stay identical across the three runs so the comparison isolates effort alone?

Keep the model fixed at claude-opus-5 and only change output_config.effort between runs - same prompts, same order, same max_tokens.

for effort in ["low", "medium", "high"]:
    correct = 0
    for case in eval_cases:
        r = client.messages.create(
            model="claude-opus-5", max_tokens=50,
            output_config={"effort": effort},
            messages=[{"role": "user", "content": case["prompt"]}],
        )
        text = next(b.text for b in r.content if b.type == "text").strip().lower()
        correct += case["expected"] in text
    print(effort, correct / len(eval_cases))

Write a function that computes cost per completed task - total dollar cost across all results, divided by the number of correct results, with a penalty added for each incorrect case to represent a retry.

This operationalises 'cost per completed task, not cost per token' - a config with a lower per-call cost but a lower accuracy can have a worse cost-per-completed-task number once failures are counted.

You should see: A single dollar figure per configuration that accounts for both spend and failure rate, not just raw token cost.

Hints

If a case comes back wrong, what real-world cost does that represent, and how would you add it into the total?

Sum the dollar cost of every call, add a fixed retry_cost for each incorrect result, then divide by the count of correct results (not the total case count) to get cost per completed task.

def cost_per_completed_task(results, input_rate, output_rate, retry_cost=0):
    total_cost = sum(
        r["usage"].input_tokens / 1e6 * input_rate + r["usage"].output_tokens / 1e6 * output_rate
        for r in results
    )
    completed = sum(r["correct"] for r in results)
    failed = len(results) - completed
    total_cost += failed * retry_cost
    return total_cost / max(completed, 1)

Run the Claude Sonnet 5 eval from step 2 five times and compute the mean and standard deviation of the accuracy across trials.

This demonstrates why a single comparison is not a valid basis for a keep/revert decision - the exam trap about judging model fit from one run is directly addressed by seeing the spread across repeated trials.

You should see: A mean accuracy close to what you saw in step 2, with a nonzero standard deviation showing run-to-run variation.

Hints

What Python module gives you mean and standard deviation without writing the formulas yourself?

Use the statistics module: run the eval in a loop 5 times, collect the accuracy score from each run, then compute statistics.mean and statistics.pstdev on the list.

import statistics
trial_scores = []
for trial in range(5):
    results = run_eval("claude-sonnet-5", eval_cases)
    trial_scores.append(sum(r["correct"] for r in results) / len(results))
print("mean:", statistics.mean(trial_scores), "stdev:", statistics.pstdev(trial_scores))

Sources

Anthropic docs - Choosing a model