Integration · Lesson 4 of 8

1.4 - Observability Challenges at Scale and Choosing a Monitoring Strategy

Decide what to log and trace across model calls, tool calls and multi-agent runs, and choose sampling, correlation and privacy controls that keep observability affordable and safe at scale.

Conventional monitoring watches for crashes, error rates and latency. LLM and agent systems add failures that all of those can miss: a 200 OK carrying a wrong or incomplete answer, a tool loop that burns tokens, a subagent that quietly covered half the scope. Agents also make dynamic decisions and can behave differently across runs with the same prompt, so you cannot reproduce a bad run by re-sending the input. Observability here means recording enough at run time to reconstruct what happened, at a cost and privacy footprint you can defend. Ongoing quality monitoring and alerting belong to Lesson 3.6; this lesson is about what to instrument and how.

What to capture, layer by layer

Layer	Capture	Why it matters
Request	Anthropic `request-id`, model, key parameters (effort, `max_tokens`, `tool_choice`), token usage, latency (time to first token and total), `stop_reason`, error type	The request ID is what support needs; usage and latency drive cost and SLOs; stop reasons expose truncation and tool-use turns
Tool	Tool name, redacted arguments, duration, success or error, result size, permission decision	Shows loops, slow dependencies and what the agent was actually allowed to do
Agent trace	One trace per user task linking model calls, tool calls and subagent runs	The only view that explains a multi-step outcome
Outcome	Task success, user feedback, escalation, linked to the trace ID	Turns telemetry into quality signal
Cost and usage	Aggregates by model, workspace, key	Budgeting, chargeback, anomaly detection

Details worth knowing. Every API response carries a request-id header, repeated as request_id in error bodies, and the SDKs expose it (for example _request_id in Python). Log it on every call. For streaming, an error can arrive after the API has already returned a 200, so status-code monitoring alone misses mid-stream failures; log the stream's error events too. The Usage and Cost Admin API reports token and cost data by model, workspace, API key and service tier in minute, hour or day buckets, typically within about five minutes, and needs an Admin credential. It is the right source for spend and chargeback, but it is aggregated: it cannot tell you why one request failed.

Traces are the unit for agents

Claude Code and the Agent SDK show the target shape. They export OpenTelemetry metrics, log events and (in beta, behind CLAUDE_CODE_ENHANCED_TELEMETRY_BETA=1) traces: a claude_code.interaction span per prompt, with claude_code.llm_request and claude_code.tool child spans. When an agent spawns a subagent, the subagent's spans nest under the parent's tool span, so the whole delegation chain reads as one trace. The SDK also propagates W3C trace context (TRACEPARENT), so an agent run appears inside your application's own trace rather than as an orphan. Resource attributes such as OTEL_SERVICE_NAME and OTEL_RESOURCE_ATTRIBUTES let you tag agent name, version, environment and, if you percent-encode them, the end user on whose behalf it acted.

If you build orchestration on the Messages API instead, reproduce the pattern deliberately. Subagents receive only what you pass them, so correlation does not happen by itself: create the trace ID in the coordinator, attach it to every model call, tool call and subagent invocation through your own span context, and record parent-child links. A per-subagent view that cannot be joined back to the coordinator's decomposition hides the failures Lesson 2.4 warns about.

Sampling, volume and cardinality at scale

Aggregate metrics are cheap and unsampled; traces and content are the expensive part. Sampling them uniformly (say 1%) is simple but drops precisely the rare failures you want. A common engineering pattern is tail-based: decide after the trace completes and keep all errors, slow traces, high-cost traces and low-feedback outcomes, sampling the healthy remainder. This is an engineering practice, not an Anthropic requirement.
Cardinality. Session IDs, user IDs and prompt hashes as metric labels multiply your time series. Keep them on traces and events. Claude Code exposes switches such as OTEL_METRICS_INCLUDE_SESSION_ID for this reason.
Body size. Full request and response bodies contain the whole conversation history. Claude Code's raw-body export is opt-in and truncates inline bodies at 60 KB by default, with an option to write full bodies to files. Decide deliberately what is worth storing.
Export reliability. Telemetry export must never block the agent. The Claude Code CLI drops data silently on export errors unless you enable diagnostics, and short-lived processes can lose buffered spans if killed before a flush, so shorten export intervals for short tasks and verify data actually arrives.

Key concept: metadata by default, content by exception

Model, tokens, latency, tool names, stop reasons, IDs and outcomes are usually safe to keep broadly. Prompt text, tool arguments, file contents and raw bodies are data-handling decisions. Claude Code follows this: by default it exports only structural metadata, and content requires explicit opt-in variables (OTEL_LOG_USER_PROMPTS, OTEL_LOG_TOOL_DETAILS, OTEL_LOG_TOOL_CONTENT, OTEL_LOG_RAW_API_BODIES).

Privacy of what you log

Logs of prompts and tool results can hold customer data, credentials pasted by users and confidential documents, and a log store is often more widely readable than the source system. Design for it: capture content only in environments and for cohorts where you have approval, redact identifiers before writing, give content a shorter retention and tighter access than metadata, and hold sampled failure traces behind a break-glass access path. Anthropic's write-up of its own multi-agent research system describes monitoring high-level interaction patterns without inspecting individual conversation content. Anthropic-side retention is a separate question from your own logs; feature pages state whether zero data retention applies. This is architecture guidance, not legal advice; regulatory obligations are covered in Lesson 4.4.

Common exam distractor

"Log every prompt and response in full so any incident can be replayed" sounds thorough but is a privacy and cost failure. "Status codes and latency dashboards are enough" misses wrong answers behind a 200 and mid-stream errors. "Use the Usage and Cost API to trace a failing request" confuses aggregate billing data with request-level traces. The strong answer correlates a trace across agents and keeps content behind explicit, governed opt-ins.

Exam traps

Relying on HTTP status codes, error rates and latency dashboards as sufficient observability for an agent

Failures in LLM systems often return 200 with a wrong or partial answer, and streaming errors can occur after a 200. You need request IDs, stop reasons, tool traces and outcome signals linked by a trace ID.
Logging complete prompts, tool results and API bodies for every request so anything can be debugged later

That creates a large store of sensitive data with its own breach, retention and compliance risk, and large storage cost. Default to metadata, make content an explicit governed opt-in, and keep full detail only for sampled failures under access control.
Sampling 1% of traces uniformly to control cost

Uniform sampling discards most of the rare failing and slow traces you need. Keep aggregate metrics unsampled and use tail-based rules that retain errors, slow, costly and low-feedback traces while sampling the healthy majority.
Using the Usage and Cost API as the tool for diagnosing why a specific agent run failed

It reports aggregated token and cost data by model, workspace, key and tier, with a short delay. It supports budgeting and chargeback but has no per-request, per-tool or per-subagent detail.

Practice question

A company runs a coordinator agent that delegates to four subagents. About 5% of runs return subtly wrong answers that nobody can reproduce. Today the team logs only the HTTP status and the final answer, and the privacy office forbids storing raw customer content in observability systems. Which approach best supports diagnosing these failures?

A Turn on full raw request and response logging for all runs, keep it for 90 days with access limited to the platform team, and search the logs by timestamp and customer name when a failure is reported
It violates the privacy constraint, is expensive at scale, and still lacks a way to join the coordinator and subagent records into one run unless correlation IDs are added.
B Use the Usage and Cost API to compare per-run token counts and infer which subagent went wrong, then add alerts on any subagent whose token use spikes
It provides aggregates by model, workspace and key, not per-run or per-subagent traces, so it cannot reconstruct why a specific run produced a wrong answer.
C Emit one correlated trace per task spanning the coordinator, its tool calls and every subagent, record metadata by default, and keep redacted content only for tail-sampled failing traces under access control Correct
It gives a joinable, reconstructable record of the coordinator's decomposition and each subagent's actions, respects the privacy rule by defaulting to metadata, and spends storage where the failures are.
D Sample 1% of runs uniformly and store their full transcripts, so a representative slice can be studied for the causes of the wrong answers
A 1% uniform sample will contain few of a 5% failure population's runs, still stores full content against the privacy rule, and lacks the correlation across agents.

Build exercise: Instrument a multi-agent flow with correlated, privacy-aware telemetry

Intermediate · 90 minutes

You'll practice:

Choose what to record at request, tool, agent and outcome layers
Propagate a correlation ID through a coordinator and its subagents
Design tail-based sampling and control metric cardinality
Separate metadata from content and define opt-in, retention and access rules
Verify that telemetry lets you reconstruct a failing run

Write a telemetry spec for a coordinator with three subagents: the fields you will record at the request, tool, agent-trace and outcome layers, and which of them are metadata versus content.

A spec decided in advance prevents both under-logging (nothing to debug with) and over-logging (a privacy liability).

You should see: A table with columns: layer, field, metadata or content, retention, who may read it.

Hints

If you could keep only ten fields per model call, which would let you explain a slow, expensive or wrong run?

Start from: trace id, parent span id, request id, model, effort, input and output tokens, latency, stop_reason, tool names. Mark prompt text, tool arguments and results as content.

Example rows: request | request_id | metadata | 30 days | on-call. request | prompt text | content | 7 days, redacted | break-glass only. tool | tool name, duration, status | metadata | 30 days | on-call.

Implement a thin wrapper around your model call that emits one structured record per call with the request ID, usage, latency, stop reason and tool names, plus the trace and span IDs passed in.

The wrapper is where correlation and the metadata default are enforced in code, so individual call sites cannot forget them.

You should see: One JSON line per model call that contains a trace_id, span_id, request_id and token counts, and no prompt text.

Hints

How will the subagent's calls end up with the same trace_id as the coordinator's?

Pass a context object (trace_id, parent_span_id) into every function that calls the model, and generate a new span_id per call. Read the request ID from the response object.

import json, time, uuid, anthropic
client = anthropic.Anthropic()
def traced(trace_id, parent, name, **kw):
    span, t0 = uuid.uuid4().hex[:8], time.perf_counter()
    r = client.messages.create(**kw)
    print(json.dumps({'trace': trace_id, 'span': span, 'parent': parent, 'name': name, 'request_id': r._request_id, 'model': kw['model'], 'in': r.usage.input_tokens, 'out': r.usage.output_tokens, 'stop': r.stop_reason, 'secs': round(time.perf_counter() - t0, 3), 'tools': [b.name for b in r.content if b.type == 'tool_use']}))
    return r

Run the coordinator on five tasks and reconstruct one run from the records alone: list the subagents that ran, the order of calls, tokens per subagent and the tool calls made.

The reconstruction test is the acceptance test for observability: if you cannot rebuild the run, you cannot debug it.

You should see: A timeline for one trace showing coordinator and subagent spans nested by parent, with token and latency totals per subagent.
Hints
1. ```
Can you tell from the records which subtopics the coordinator assigned, without any prompt text?
```
2. ```
Add a small metadata field to the coordinator's span for its decomposition, such as a list of subtopic labels, rather than storing full prompts.
```
3. ```
Group records by trace, sort by start time, and print an indented tree using the parent field. If a subagent record has no parent, correlation is broken.
```
Design the sampling and retention policy: which traces you always keep, what you sample, what stays out of metrics labels, and how long each tier is retained.

Cost and cardinality are what break observability at scale, and uniform sampling drops the traces you most need.

You should see: A short policy with keep rules (errors, slow, high-cost, negative feedback), a sampling rate for the rest, and a list of fields that are trace-only rather than metric labels.
Hints
1. ```
Which single rule ensures a rare failure is retained even at a very low sample rate?
```
2. ```
Decide after the trace ends: keep if error or stop_reason abnormal, latency over threshold, cost over threshold or user flagged; otherwise sample a small percentage.
```
3. ```
Example: keep 100% of traces with an error, a user thumbs-down, latency over p99 or cost over 3x median; sample 2% of the rest. Never use user_id or session_id as a metric label; keep them on traces.
```

Write the privacy addendum: which content fields may be captured, in which environments, with what redaction, retention and access, and what approval is needed to switch content logging on.

Content capture is a data-handling decision that observability teams routinely get wrong by default.

You should see: A one-page addendum a privacy reviewer could approve, including a default of metadata only and a documented opt-in path.

Hints

Who can read a full failing transcript, and how is that access logged?

Default off in production. Enable only for a named cohort, redact known identifier patterns before write, use a short retention, restrict reads to a break-glass role and audit each read.

State: metadata-only by default; content capture requires a ticket approved by the data owner, redaction of emails and account numbers at the collector, 7-day retention, break-glass read access with an audit log. Note it is design guidance and not legal advice.

Sources

Monitoring Claude Code usage with OpenTelemetry (Claude Code docs) - Anthropic
Observability with OpenTelemetry in the Agent SDK (Claude Code docs) - Anthropic
API errors and request IDs (Claude API docs) - Anthropic
Usage and Cost API (Claude API docs) - Anthropic
How we built our multi-agent research system (Anthropic Engineering) - Anthropic