Evaluation, Testing & Optimisation · Lesson 6 of 6

3.6 - Production Monitoring with Logging and Observability Tooling

Design ongoing monitoring for a live Claude system: which quality, cost, latency, error and safety signals to track, how to detect drift and regressions, how to alert sensibly, and how to sample for human review.

Pre-release evaluation tells you a system was good on the day you tested it. Production monitoring tells you whether it is still good. Claude systems degrade in ways that ordinary services do not: the HTTP status stays 200 while answers get worse because the input mix shifted, a retrieval index was rebuilt, a tool changed its output, or a prompt edit quietly harmed one slice of traffic. Lesson 1.4 covers what to log and trace across an LLM, agent and tool integration, including correlation across multi-agent runs and the privacy of logged content. This lesson takes those logs as given and asks a different question: what do you watch, what do you alert on, and how do you catch quality decay early enough to matter?

Anthropic’s agent-evals write-up positions monitoring as one layer among several. It reveals real user behaviour at scale and catches what synthetic evals miss, but it is reactive: problems reach users before you know. Automated evals give fast iteration before release, and user feedback is sparse, self-selected and skews toward severe issues. A monitoring design should say which failures it is expected to catch, and which it is not.

Three families of signals

Family	Signals	Where they come from	Alert on
Operational	Time to first token and total latency percentiles; error rate by status and `error.type`; retries; timeouts; share of responses that stopped at `max_tokens`	Your request log and response fields; the `request-id` header	Percentile shifts against baseline; sustained error-rate rise; 429 and 529 treated separately; truncation spikes
Economic	Input, output, cache-read and cache-write tokens; cache-read share; cost per completed task; batch share; spend by model and workspace	The `usage` object per call; the Usage and Cost Admin API; Console Usage and Cost pages	Cost per task drift; cache-read share drop; spend approaching a limit
Quality and safety	Sampled judge scores; groundedness or citation-check failures; guardrail and injection-screen trigger rates; refusal and escalation rates; edit or thumbs-down rates; task-completion proxies	Online scoring of sampled traffic; guardrail logs; product analytics; human review	Score drop on a slice; rise in guardrail triggers; jump in escalations or corrections

Some notes for architects. The Usage and Cost Admin API needs Admin API credentials, and usage typically appears within about five minutes; daily cost buckets and per-request logs answer different questions. Anthropic lists observability partners (Datadog, Grafana Cloud, Honeycomb and others) for dashboards and alerts on usage and cost. Neither replaces your own per-request log, because only that log can join tokens and latency to task outcome and prompt version.

Instrument so that every signal can be explained

For each call record a stable set of fields: request id, timestamp, model ID, prompt and tool-set version, experiment variant, effort and other parameters, latency (first token and total), the usage fields, stop_reason, error status and type, tool calls and outcome, and a task or conversation id that lets you compute per-task cost and success. Tag deploys, prompt changes, model migrations, retrieval index rebuilds and tool changes as events on the same timeline, because most regressions coincide with one of them. Logged prompts and outputs can contain personal or regulated data, so apply redaction, retention limits and access control from the start (Lessons 1.4 and 4.4).

Detecting drift and regressions

Separate the sources of change, because they need different detectors:

Input drift. Users start asking different things, or in different languages, or with longer documents. Watch input length, intent or topic mix, and the share of cases that fall outside your evaluation slices. New clusters are candidates for new eval cases (Lesson 3.2).
Your own changes. Prompts, tool definitions, retrieval indexes, chunking and upstream data change often. Every Claude model ID is a pinned snapshot, so an unchanged ID does not silently change its behaviour; model changes are deliberate migrations you should treat as experiments (Lesson 3.3). That makes your own release events the most likely regression source.
Dependency drift. A tool API changes its schema, an index goes stale, a rate limit tier changes. These show up as tool errors, retrieval misses or rising latency.

Two detectors work together. Scheduled replay runs your held-out or regression set against the live pipeline on a schedule and after every change, comparing to a stored baseline with a pre-set tolerance; it is deterministic in inputs, so movement means the system changed. Online sampling scores a sample of real traffic with a judge (different model, fixed rubric, calibrated against humans; Lesson 3.1) and tracks the score per slice over time; it catches problems your fixed set does not contain. Compare against a baseline window rather than an absolute number, use enough samples per window that noise is not mistaken for signal, and remember that non-determinism means single-run swings are expected.

Key concept: monitoring closes the evaluation loop

Monitoring is not separate from evaluation. Production failures found by sampling, feedback and alerts become new cases in the evaluation dataset, the fixed judge is re-calibrated on freshly human-labelled samples, and each fix is validated offline and online before rollout (Lessons 3.2 and 3.3). A monitoring system that never feeds back into the eval set finds the same failure repeatedly.

Alerting that people will trust

Alert on user-visible symptoms tied to your SLOs, and use dashboards for causes. Page on “p95 latency above the objective for N minutes”, not on a single slow call.
Use percentiles and baselines, not averages. Means hide tails, and a fixed threshold ignores daily patterns. Compare with the same period in prior weeks where traffic is seasonal.
Separate error classes. A 429 means your own limit or a spend cap was hit; a 529 or 500 is capacity or a fault on the provider side; a 400 is your bug. Anthropic notes that a tier spend cap can return a 429 with no retry-after header that keeps failing, so it needs its own alert rather than blind retries. The API also returns a 400 once an organisation or workspace spend limit you set is reached, so alert on spend approaching that limit before it turns into an outage. Batches may slightly exceed a workspace spend limit.
Watch truncation and format failures. A rise in max_tokens stops or unparseable outputs is an early sign of a prompt or data change.
Control noise. Tier severity, require persistence across windows, route quality alerts to a review queue and outages to on-call, and attach a runbook (Lesson 3.4 gives the diagnostic order; Lesson 3.3 gives rollback).

Sampling for human review

Humans cannot read all traffic, so sample deliberately. Use a random baseline sample to estimate overall quality without bias, plus targeted samples: low judge scores, guardrail or injection-screen triggers, negative feedback, escalations, very long or unusual inputs, new topic clusters and outputs near a decision threshold. Stratify by slice so rare but important segments are not drowned out. Size the review load to reviewer capacity (Lesson 4.3 covers reviewer load and automation bias), record reviewer labels with rubric versions, and use disagreement between reviewers and the judge to recalibrate the judge. Every confirmed failure gets a root-cause label and, where useful, becomes an eval case.

Common exam distractor

Beware answers that equate health with availability (“the API returns 200 and error rates are flat”), alert on mean latency, rely only on user thumbs-up or thumbs-down as the quality signal, or treat production monitoring as a replacement for pre-release evaluation. Quality drift can occur with every technical metric green; the exam favours sampled quality scoring, scheduled replay of a regression set, baseline-relative alerts, and monitoring layered on top of offline evals.

Exam traps

Declaring the system healthy because latency, throughput and HTTP error rates are normal.

A Claude system can return successful responses whose quality has degraded after an input shift, a retrieval change or a prompt edit. Quality must be measured directly through sampled scoring, scheduled replay of a regression set, feedback and human review.
Alerting on mean latency or a single fixed threshold for all times of day.

Averages hide tail latency, which is what breaches SLAs and frustrates users. Percentiles measured against a baseline window, with persistence requirements, give alerts people can act on.
Using user thumbs-up and thumbs-down ratings as the primary quality signal.

Feedback is sparse and self-selected, skews toward severe problems, and rarely says why something failed. It is a useful trigger for targeted review, but it needs a random sample and calibrated automated scoring alongside it.
Relying on production monitoring instead of pre-release evaluation, or the reverse.

Monitoring is reactive: problems reach users before you know. Offline evals and staged rollouts prevent regressions, while monitoring catches what fixed test sets miss. Each layer covers the other's blind spot.

Practice question

Two weeks after a scheduled re-ingestion of a company's knowledge base, a Claude support assistant's escalation-to-human rate rises slowly and complaints mention outdated answers. Dashboards for latency, error rate and token spend are all normal, and no prompt or model changes were deployed. Which monitoring capability would most directly have caught the regression earlier?

A A tighter alert on 5xx error rate and average latency so that slower responses are noticed sooner, with paging as soon as either metric crosses a lower threshold for more than a minute or two.
Latency and error metrics were normal. This regression is a quality problem in returned content, which operational alerts do not measure.
B A daily total of tokens consumed, since a knowledge-base problem would increase spend, with an alert whenever the daily total moves outside its usual range by more than a set percentage.
Token spend was normal and is not a reliable indicator of answer correctness. A stale index need not change token volume at all.
C Collecting thumbs-up and thumbs-down feedback and paging when the daily count of thumbs-down passes a fixed number, so that user complaints reach the team quickly and can be triaged by hand.
Feedback is sparse, self-selected and lagging, and a fixed count ignores traffic changes. It could help as one trigger for review but would detect the problem late and without diagnosing it.
D Scheduled replay of a regression set against the live pipeline after each ingestion, plus sampled online scoring of groundedness per slice, with baseline-relative alerts and an annotation for the ingestion event. Correct
Replay after the ingestion event would show the score drop immediately, sampled scoring would detect it on real traffic, and annotating pipeline events ties the change to its cause.

Build exercise: Build a small quality-and-cost monitoring loop: log, roll up, replay, alert and sample

Intermediate · 80 minutes

You'll practice:

Define signals, baselines and alert rules across operational, economic and quality families
Compute percentile latency, error mix, truncation rate and cache-read share from a per-call log
Implement a scheduled regression replay that compares to a baseline with a tolerance
Select a stratified sample for human review and route confirmed failures into the eval dataset

For a Claude application you know, write a monitoring specification: at least eight signals across the operational, economic and quality families, and for each the source, the baseline you will establish, the alert condition (percentile, rate or baseline-relative), the severity and the first diagnostic step. Include a separate rule for 429s and one for spend approaching a limit.

Writing the alert conditions and runbook step before an incident is what makes alerts actionable, and the exam expects you to distinguish error classes and to alert on symptoms.

You should see: A table with at least eight rows in which no alert relies on a mean, every row names its source, and quality signals are present, not only latency and errors.
Hints
1. ```
Which of your signals would still look normal if the retrieval index were silently stale?
```
2. ```
Make sure at least one row is quality-based (sampled judge score or replay pass rate) and one is economic (cache-read share or cost per completed task).
```
3. ```
Example row: Signal: replay pass rate on the regression set. Source: nightly job against the live pipeline. Baseline: mean of the last stable releases. Alert: drops more than the agreed tolerance below baseline, or any gate slice fails. Severity: high, blocks the next rollout. First step: check the release and ingestion event annotations, then follow the diagnostic tree in Lesson 3.4.
```

Extend your Claude call wrapper so every call appends a JSON line with timestamp, request id, model, prompt version, latency, status, stop_reason and the usage fields, plus a task id. Generate at least 200 log lines by running your app or a synthetic driver, including a few failures.

Everything else in the loop depends on a consistent per-call record; it is also the only place that can join tokens and latency to a task outcome.

You should see: A calls.jsonl file with complete records for successes and failures, each carrying a request id and prompt version.

Hints

Which fields would you need in order to explain a p95 spike after the fact, and which do you get for free from the response object?

Time the call yourself, take request id, stop_reason and usage from the response, and on an API error record the status code and the request-id header.

import json, time, anthropic
client = anthropic.Anthropic()

def monitored_call(task_id, prompt_version, **request):
    rec = {"ts": time.time(), "task_id": task_id, "prompt_version": prompt_version, "model": request["model"]}
    t0 = time.perf_counter()
    try:
        r = client.messages.create(**request)
        u = r.usage
        rec.update(ok=True, request_id=r._request_id, stop_reason=r.stop_reason,
                   input=u.input_tokens, output=u.output_tokens,
                   cache_read=u.cache_read_input_tokens or 0, cache_write=u.cache_creation_input_tokens or 0)
        return r
    except anthropic.APIStatusError as e:
        rec.update(ok=False, status=e.status_code, request_id=e.response.headers.get("request-id"))
        raise
    finally:
        rec["seconds"] = time.perf_counter() - t0
        open("calls.jsonl", "a").write(json.dumps(rec) + "\n")

Write a rollup script that groups the log by hour and reports p50 and p95 latency, error rate by status, the share of stop_reason equal to max_tokens, cache-read share of input tokens and tokens per task. Compare each hour with a baseline computed from earlier hours and print an alert line when p95 latency or truncation rate exceeds the baseline by a factor you choose.

Percentile and share metrics with a baseline comparison are the building blocks of alerts that do not fire on noise.

You should see: An hourly table and at least one alert line produced by injecting a slow or truncated batch of calls into the log.

Hints

What baseline window would you use, and how would you avoid comparing a busy weekday hour with a quiet night?

Group by hour, compute the metrics per group, and compare with the median of the earlier groups. The factor is an example to tune from your own data, not a recommendation.

import json, math, collections, statistics
def pct(v, p):
    s = sorted(v); return s[max(0, math.ceil(p / 100 * len(s)) - 1)]

rows = [json.loads(l) for l in open("calls.jsonl")]
by_hour = collections.defaultdict(list)
for r in rows: by_hour[int(r["ts"] // 3600)].append(r)

history = []
for hour, rs in sorted(by_hour.items()):
    p95 = pct([r["seconds"] for r in rs], 95)
    trunc = sum(r.get("stop_reason") == "max_tokens" for r in rs) / len(rs)
    total_in = sum(r.get("input", 0) + r.get("cache_read", 0) + r.get("cache_write", 0) for r in rs)
    cache_share = sum(r.get("cache_read", 0) for r in rs) / total_in if total_in else 0
    base = statistics.median(history) if history else None
    if base and p95 > 1.5 * base:      # 1.5 is an example factor; tune from your data
        print("ALERT p95", hour, round(p95, 2), "vs baseline", round(base, 2))
    history.append(p95)
    print(hour, "p95", round(p95, 2), "trunc", round(trunc, 3), "cache_read_share", round(cache_share, 3))

Create a nightly replay: run your held-out or regression set against the live pipeline, grade it with your fixed graders, compare the pass rate per slice with a stored baseline, and emit an alert if any gate slice falls below the baseline minus a tolerance. Then simulate a regression (for example by altering the retrieved context or a prompt line) and confirm the replay catches it.

Replay is the deterministic detector: inputs are fixed, so a score change means the system changed. It also proves your alert wiring works before a real incident.

You should see: A stored baseline file, a replay run that passes on the unchanged system, and an alert that fires with the failing slice named after you introduce the regression.

Hints

If the replay passes but production quality complaints continue, what does that tell you about the coverage of the replay set?

Store per-slice baseline pass rates with the dataset hash, compare each slice separately, and fail loudly on any gate slice.

baseline = json.load(open("baseline.json"))     # {"dataset_hash": "...", "slices": {"representative": 0.93, "adversarial": 0.9, ...}}
TOL = 0.05                                       # example tolerance; set from run-to-run variance you measured
current = run_regression_set()                   # returns {"slice": pass_rate}
for name, base in baseline["slices"].items():
    if current[name] < base - TOL:
        print(f"ALERT replay regression in slice '{name}': {current[name]:.2f} vs baseline {base:.2f}")

Build a review sampler: from your log (with fields such as judge_score, guardrail_triggered and thumbs), select a random baseline sample plus targeted samples (low judge score, guardrail triggers, thumbs-down, new intent), remove duplicates, and export them for human labelling. After labelling, add each confirmed failure to your eval dataset with a slice tag and write down what changed in the judge rubric, if anything.

Deliberate sampling is how a small review budget stays informative, and adding confirmed failures to the dataset is what closes the monitoring loop.

You should see: A review file that mixes random and targeted items with the reason each was selected, and at least one new eval case created from a labelled failure.

Hints

If you only review thumbs-down cases, what will you never learn about the quality of traffic that received no feedback?

Take a random sample for an unbiased estimate, then add targeted items; record the selection reason so you can report each stratum separately.

import random
def review_sample(rows, n_random=25, seed=3):
    rng = random.Random(seed)
    picked = {r["task_id"]: dict(r, reason="random") for r in rng.sample(rows, min(n_random, len(rows)))}
    for r in rows:
        why = ("low_judge" if r.get("judge_score") is not None and r["judge_score"] <= 2 else
               "guardrail" if r.get("guardrail_triggered") else
               "thumbs_down" if r.get("thumbs") == "down" else None)
        if why and r["task_id"] not in picked:
            picked[r["task_id"]] = dict(r, reason=why)
    return list(picked.values())

Sources

Demystifying evals for AI agents (Anthropic Engineering) - Anthropic
Usage and Cost API (Claude docs) - Anthropic
Claude API errors (Claude docs) - Anthropic
Mitigate jailbreaks and prompt injections: continuous monitoring (Claude docs) - Anthropic
Models overview (Claude docs) - Anthropic
Batch processing (Claude docs) - Anthropic