Governance, Safety & Risk Management · Lesson 3 of 5

4.3 - Human-in-the-Loop Validation Strategies

Decide where human review belongs, when to use approval gates, sampling or confidence-based routing, and how to size reviewer capacity and defend against rubber-stamping and automation bias.

Human-in-the-loop (HITL) is a control with its own cost and its own failure modes, so it has to be designed like any other. Four questions structure the design: where in the flow a human sits, how they participate (blocking gate, after-the-fact sampling, or exception handling), which items reach them, and whether reviewers can keep up and stay independent. Scenario questions in this area almost always hinge on one of these four being answered badly.

Where review belongs: consequence, reversibility and obligation

Two properties decide whether an action deserves a human before it takes effect: the consequence of a wrong instance (money, safety, legal standing, data exposure) and its reversibility (a draft in a queue is cheap to catch; a sent email, an executed transfer or a notified applicant is not). High consequence plus low reversibility is a blocking gate. Low consequence or easily reversed work is automated, with review moved to sampling. Two further inputs are external obligations and agent uncertainty.

Obligations. Anthropic's Usage Policy sets extra requirements for high-risk use cases (it names legal, healthcare, insurance, finance, employment, housing, academic testing and media): when you give advice, recommendations or subjective decisions that directly affect individuals or consumers, a qualified professional in the field must review the content or decision prior to dissemination or finalization, and when outputs are presented directly to individuals or consumers you must disclose that AI is helping to produce them. Separately, the European Commission's GDPR guidance says people should not be subject to decisions based solely on automated processing that have legal or similarly significant effects, with narrow exceptions (necessity for a contract, authorisation by law with safeguards, explicit consent) and safeguards that include a right to human review and to contest. Whether a given system falls under these is a legal determination (Lesson 4.4); the architect's part is to make meaningful review a real, evidenced capability of the workflow.
Agent blockers. Anthropic's guidance on agents describes pausing for human feedback at checkpoints or when the agent hits a blocker, with stopping conditions such as an iteration limit to keep control.

A review only counts if the reviewer is qualified to judge the specific content, has the authority to change the outcome, sees it before it takes effect, and is checking for the risk that actually applies (a fabricated citation, a biased rationale), not skimming for typos.

Gates, sampling and confidence-based routing

Approval gate. Every item is reviewed before it takes effect. Full coverage, but capacity equals volume, latency is added, and a gate on everything breeds approval fatigue. Reserve it for the high-consequence, hard-to-reverse tier, and use tiered thresholds (auto below a limit, any reviewer in the middle, a senior reviewer above) so friction tracks risk.

Sampling. A random or stratified sample of already-automated outputs is reviewed afterwards. It measures quality and detects novel error patterns, but it does not stop an individual bad output from taking effect. It suits reversible or low-stakes work, and it is the necessary complement to routing, never a substitute for a gate on irreversible actions.

Confidence-based routing. Automate what the system is sure about, send the rest to humans. Its value depends entirely on the quality of the uncertainty signal:

A confidence figure the model writes is generated text, not a calibrated probability. It becomes usable only after you compare it with ground truth on a labelled set and derive thresholds per segment (document type and field), because the same stated score can mean different accuracy in different segments.
Prefer behavioural and deterministic signals where you can: a schema or business-rule failure, a quoted source passage that cannot be found in the document, disagreement across repeated runs (the best-of-N technique from Anthropic's hallucination guidance), out-of-distribution input, or a high monetary value.
Beware the aggregate trap. As a purely hypothetical illustration, a pipeline reporting high overall accuracy can still perform badly on a small segment such as handwritten or non-standard documents, because the common segment dominates the average. Validate by segment before you automate anything.

Sequence matters: measure accuracy per segment, calibrate, set thresholds, run stratified random sampling that includes the automated high-confidence stratum, and only then reduce human review for segments with consistently validated accuracy. If you only ever look at low-confidence items you are reviewing what the system already flagged and learning nothing about the errors it is confidently making.

Common exam distractor

Expect answers that sound prudent but fail on one axis: review every item (unsustainable, causes rubber-stamping); sample only for an irreversible high-consequence action (detects errors after harm); trust the raw confidence score or the aggregate accuracy figure; have another model review the first model's output and remove humans (correlated errors, and no accountable person); and review after the applicant is notified (quality assurance, not oversight). The right answer pairs a pre-effect gate for the risky tier with calibrated routing and stratified sampling for the rest.

Mechanics: pausing an agent for a human

In the Agent SDK, a canUseTool callback is invoked when nothing earlier in the permission flow has approved the call; execution stays paused until it returns, and it can stay pending indefinitely. It returns allow (optionally with modified input, so the reviewer can approve with changes) or deny with a message that Claude sees and can act on. Calls auto-approved earlier never reach it, so a must-run review belongs in a PreToolUse hook (Lesson 4.1). If the human may take longer than your process should stay alive, a PreToolUse hook can return the defer decision so the process exits and the session resumes later from its persisted state. That is the pattern for approvals that take hours, such as a manager sign-off.

Reviewer load and automation bias

A review step is only as good as the attention behind it. Do the capacity arithmetic first: items per day requiring review multiplied by realistic minutes per item must fit inside reviewer hours with headroom. If it does not, the queue grows and delays the business effect, or reviewers speed up and the gate degrades into a rubber stamp. Fixes are structural: narrow the gate to the truly high-consequence tier, add tiers, order the queue by uncertainty (highest first, not arrival order), and improve upstream accuracy so fewer items need attention.

Automation bias is the tendency to accept a confident suggestion, and fluent model output invites it. Countermeasures that work at the system level: show the evidence next to the conclusion (the source passage, the quote, a diff of what changed) rather than the conclusion alone; for a sample, capture the reviewer's own judgement before revealing the model's; seed the queue with known-bad canary items and measure how often reviewers catch them; and watch indicators such as approval rate, time per item and override rate. An approval rate drifting towards 100% with shrinking review time is a warning sign worth investigating, though not proof of failure (the model may simply be good). Feed reviewer corrections back into your evaluation set so the review effort improves the system instead of only patching outputs.

Key concept

Match the oversight mechanism to the risk. Blocking, pre-effect gates for high-consequence and hard-to-reverse actions; calibrated, per-segment confidence routing for the middle; stratified sampling that includes the automated stratum to measure and to catch new error patterns. Then protect the human step itself: capacity, evidence, canaries and metrics, or the gate becomes decoration.

Exam traps

Requiring human approval for every tool call or every output so that nothing is unreviewed.

Uniform gating exceeds reviewer capacity and trains approvers to click through. Reserve blocking review for high-consequence, hard-to-reverse actions and use tiers, routing and sampling elsewhere.
Reviewing only the low-confidence items and treating the high-confidence items as safe because the system reports high accuracy.

Low-confidence items are already routed to humans, so reviewing them cannot reveal errors in the automated stratum. Stratified random sampling of high-confidence items is what detects confidently wrong output and new error patterns.
Using the model's stated confidence, or an overall accuracy figure, as the threshold for removing human review.

Self-reported confidence is not a calibrated probability, and an aggregate can hide a weak segment. Calibrate against labelled data per segment before setting thresholds and reducing review.
Meeting an oversight requirement with a spot-check of decisions that have already been sent to the affected people.

Oversight must occur before the decision takes effect, by someone qualified who can change it. After-the-fact sampling is useful quality assurance but leaves most decisions unreviewed at the time they affect a person.

Practice question

An insurer processes a very large volume of claims. The proposal: auto-approve claims where the model's stated confidence exceeds a fixed threshold, auto-send denials to policyholders, and have QA staff review a random 2% of the low-confidence claims each week. Which redesign best addresses the weaknesses?

A Raise the confidence threshold so that only near-certain claims are auto-approved, keep denials auto-sent, and keep the weekly 2% sample of low-confidence claims as the quality check.
A higher threshold on an uncalibrated score is still an uncalibrated threshold, and it leaves denials unreviewed and the automated stratum unmeasured.
B Send every claim to a human adjuster before any decision is issued, so that no automated decision ever reaches a policyholder, and hire enough adjusters to keep up with claim volume.
This removes the value of automation, likely exceeds reviewer capacity, and invites rubber-stamping. Proportionate gating protects the high-consequence tier while letting validated segments flow.
C Measure accuracy per claim type and calibrate confidence against outcomes; route denials to a qualified adjuster before notification; stratified-sample the auto-approved claims too. Correct
It fixes all three flaws: uncalibrated confidence and aggregate blindness (per-segment calibration), a consequential decision going out with no review (pre-notification adjuster gate), and no visibility into automated decisions (stratified sampling that includes the high-confidence stratum).
D Have a second model instance re-review all automated approvals and denials, and drop human review and sampling in favour of the model-to-model check, since it scales to every claim.
A second pass by a similar model can share the first one's blind spots, and no accountable person is positioned to judge or stand behind an adverse decision. It can be a useful additional signal but not a replacement for human oversight.

Build exercise: Design and simulate a review policy with routing, sampling and reviewer-vigilance metrics

Advanced · 75 minutes

You'll practice:

Classify actions into gate, route or sample tiers using consequence and reversibility
Demonstrate why aggregate accuracy hides weak segments and derive per-segment thresholds
Implement a router that combines deterministic signals with calibrated confidence
Implement stratified sampling that includes the automated stratum, and measure reviewer vigilance with canaries

Write an action inventory for a claims-triage workflow (summarise claim, draft decision letter, approve claim under a limit, deny claim, send letter to policyholder). For each action record consequence, reversibility, any external obligation, and your chosen oversight mode: blocking gate, calibrated routing, or sampling.

The inventory is the design. If you cannot justify a mode from consequence and reversibility, you will end up gating everything or nothing.

You should see: A table in which sending the letter and denying a claim are blocking gates with a qualified reviewer, drafting is automated, and approval under a limit uses routing plus sampling.
Hints
1. ```
Which of these actions cannot be taken back once the policyholder has been notified?
```
2. ```
Give each action a consequence rating and a reversible yes/no. Anything high and irreversible is a pre-effect gate; anything reversible and low is sampling.
```
3. ```
Draft letter: reversible, automate (sample). Approve under limit: medium, reversible via reopening, so calibrated routing plus stratified sampling. Deny claim and send letter: high and hard to reverse, so a qualified adjuster approves before sending, with the evidence shown beside the conclusion.
```

Build a synthetic labelled dataset of at least 500 items across three segments with different true accuracy (make one segment small and poor). Compute overall accuracy and per-segment accuracy, and show that the aggregate looks healthy while the small segment does not.

Seeing the aggregate trap in your own numbers is what makes per-segment validation a habit. The data is synthetic and illustrative, so state that clearly when reporting results.

You should see: An overall accuracy that looks high alongside a small segment with clearly unacceptable accuracy, printed in the same table.

Hints

What proportion of the data must each segment have for the average to hide the weak one?

Generate items with a segment label, a boolean correct flag drawn from a segment-specific probability, and a simulated confidence. Group by segment.

import random
rnd = random.Random(7)
spec = {'standard': (0.90, 0.99), 'scanned': (0.06, 0.80), 'handwritten': (0.04, 0.55)}  # (share, accuracy) - illustrative
items = []
for seg, (share, acc) in spec.items():
    for _ in range(int(1000 * share)):
        items.append({'segment': seg, 'correct': rnd.random() < acc, 'conf': min(0.99, rnd.gauss(0.9, 0.05))})

def acc(rows):
    return sum(r['correct'] for r in rows) / len(rows)

print('overall', round(acc(items), 3))
for seg in spec:
    print(seg, round(acc([r for r in items if r['segment'] == seg]), 3))

Implement a router with three outputs (auto, human, senior) that uses deterministic signals first (validation failure, quote not found in source, high amount) and then a calibrated accuracy lookup per segment and confidence band. Unknown bands must default to human.

Deterministic signals are cheap and trustworthy; calibration turns the model's number into something you have measured. Defaulting unknowns to human is the fail-safe direction.

You should see: Test cases showing that a validation failure always goes to a human even at high confidence, a large amount goes to senior review, and an unseen band goes to human.

Hints

In what order should the checks run so that a hard rule can never be overridden by a high confidence score?

Check the hard rules first and return early. Build a dict from (segment, confidence band) to measured accuracy from step 2 and use .get with a default of 0.

def route(item, calib, min_acc=0.98, senior_above=10000):
    if not item['schema_ok'] or not item['quote_found']:
        return 'human'
    if item['amount'] > senior_above:
        return 'senior'
    band = round(item['conf'], 1)
    measured = calib.get((item['segment'], band), 0.0)  # unknown band -> 0.0 -> human
    return 'auto' if measured >= min_acc else 'human'

Implement stratified random sampling over the items that were routed to auto, stratified by segment, with a minimum number per stratum, and compute the observed error rate per stratum from the sampled items.

Sampling the automated stratum is how you find errors the system is confident about and detect new error patterns. A minimum per stratum stops small segments being sampled out of existence.

You should see: A sample containing items from every segment present in the auto tier, and a per-segment error estimate you could compare with your calibration table.

Hints

If a segment is 2% of volume, what does a flat 5% sample give you for it, and is that enough to say anything?

Group auto-routed items by segment, then take max(minimum, rate x size) from each group, capped at the group size.

def stratified_sample(auto_items, rate=0.05, min_per=5, seed=1):
    rnd = random.Random(seed)
    strata = {}
    for it in auto_items:
        strata.setdefault(it['segment'], []).append(it)
    sample = []
    for seg, group in strata.items():
        n = min(len(group), max(min_per, int(len(group) * rate)))
        sample += rnd.sample(group, n)
    return sample

Model reviewer load and vigilance: given arrivals per day and minutes per item, compute the reviewers needed; then inject known-bad canary items into a simulated review stream and compute the catch rate, approval rate and average time per item, flagging a review pool whose approval rate is high while its canary catch rate is low.

Capacity and vigilance are the properties that decide whether the human step is a real control. Canaries turn 'are reviewers paying attention?' into a number.

You should see: A capacity figure with headroom, and a report that separates a healthy reviewer from one who approves nearly everything, including the canaries.

Hints

If a reviewer approves 99% of items but misses most canaries, which of the two numbers should you believe?

Mark a small share of the queue as canaries with a known correct answer of 'reject', then compute the catch rate as rejected canaries over all canaries.

def vigilance(reviews):
    canaries = [r for r in reviews if r['is_canary']]
    caught = sum(1 for r in canaries if not r['approved'])
    return {
        'canary_catch_rate': caught / len(canaries) if canaries else None,
        'approval_rate': sum(r['approved'] for r in reviews) / len(reviews),
        'avg_seconds': sum(r['seconds'] for r in reviews) / len(reviews),
    }

reviewers_needed = (items_per_day * minutes_per_item) / (60 * productive_hours_per_reviewer)

Sources

Anthropic Usage Policy - high-risk use case requirements - Anthropic
Handle approvals and user input (Agent SDK) - Anthropic
Intercept and control agent behavior with hooks (Agent SDK) - Anthropic
Building effective agents - Anthropic
Reduce hallucinations (Claude docs) - Anthropic
Are there restrictions on the use of automated decision-making? (European Commission) - European Commission