PPassTheCertExam

Study guides / CCAR-P / Domain 4

Governance, Safety & Risk Management · Lesson 2 of 5

4.2 - Risks, Limitations and Failure Modes of LLM Systems

Identify how LLM systems fail (hallucination, injection, over-reliance, drift, non-determinism, cascading agent errors) and turn that into a risk register with controls, owners and review triggers you can defend.

Limitations, failure modes and risks are different things

Three words get blurred in scenario questions and in design reviews. A limitation is a property of the technology (it can state false things fluently, it only knows what its training and your context supply, it can be steered by text it reads). A failure mode is how that limitation shows up in your system (an invented policy exclusion in a claims summary). A risk is that failure mode weighed by likelihood and consequence in your context. The same model can be low risk in a brainstorming tool and high risk in a benefits-eligibility workflow, so the architect's job is to assess the system, not the model in the abstract.

The failure-mode catalogue an architect should carry

Hallucination. Fabricated or unsupported claims delivered in the same confident, fluent tone as correct ones. Specifics such as figures, citations, quotes and identifiers are the most dangerous because they look checkable. Anthropic's guide lists mitigations: give Claude explicit permission to say it does not know; for long documents extract word-for-word quotes first and answer only from them; require citations and have Claude retract any claim it cannot support with a quote; and use chain-of-thought verification, best-of-N comparison, iterative refinement, and restricting Claude to the provided documents. The same page is explicit that these techniques reduce hallucinations but do not eliminate them, so critical information must still be validated. The Citations feature (document blocks with citations enabled) makes the supporting passages machine-checkable. Treat hallucination as a rate you measure and bound, not a bug you fix once.

Prompt injection. Instructions hidden in content the model reads. The risk multiplies when three things coincide: untrusted input, access to sensitive data or tools, and the ability to act externally. Controls are in Lesson 4.1; for the register, the point is to record which agents have all three.

Over-reliance and automation bias. People accept fluent output and stop checking. Paradoxically the risk rises as accuracy improves: errors become rarer, harder to spot, and reviewers' attention decays. This is a socio-technical failure that no prompt fixes (see Lesson 4.3).

Drift. Behaviour changes without a code change. Sources: model version changes and retirements, shifts in the input population, edits to prompts, tools or retrieval corpora, and upstream schema changes. Anthropic's deprecation policy states that customers with active deployments get at least 60 days' notice before a publicly released model is retired, that deprecated models are likely to be less reliable than active ones, and that you should test replacements well before the retirement date. Controls: an explicit model identifier in configuration, a regression eval set, scheduled re-runs, and a tested migration path.

Non-determinism. The same input can yield different outputs. Do not design around reproducibility through sampling knobs: the docs list temperature, top_p and top_k as deprecated on Claude Opus 4.7 and later, where non-default values return a 400 error. What you can do is constrain format (structured outputs guarantee schema conformance, not correct content), evaluate distributions instead of single runs, and log inputs, outputs and the model identifier so the record of a decision is reproducible even when regeneration is not.

Cascading agent errors. Anthropic notes that agent autonomy brings higher cost and the potential for compounding errors. A wrong intermediate result becomes trusted input to the next step; a subagent's confident summary is treated as fact by the coordinator; a looping agent burns budget or repeats a harmful action. Controls: validation at every handoff (schema and semantic checks against sources), stopping conditions such as a maximum iteration count, checkpoints where the agent pauses for human feedback, idempotent tools, and sandboxed testing.

Common exam distractor

Four answers to distrust. 'Set temperature to 0' as a cure for hallucination or for unreliable results: it does not address factual errors, and the parameter is deprecated on the newest models. 'Add retrieval, so it will not hallucinate': grounding lowers the rate but retrieved content can be wrong, stale or hostile, and the model can still misstate it. 'Upgrade to the newest model' as closing a risk: a better model shifts the rate, and it also introduces drift you must re-test. 'Do the risk assessment at launch': the register must be re-scored when the model, data or scope changes.

A risk-register approach that survives review

A register is a living table, one row per risk, written as cause → event → consequence so it can be tested. Useful columns: identifier and statement; likelihood; impact; detectability; controls classified as preventive, detective or corrective and tagged with the layer they live in (Lesson 4.1); residual risk; accountable owner; and a review trigger (a model change, a new document type, an incident).

FieldExample row: claims-summary agent
StatementAgent states a policy exclusion that does not exist, so an adjuster denies a valid claim
LikelihoodMedium, from a measured rate on the eval set; varies by document type
ImpactHigh: customer harm and regulatory exposure
DetectabilityLow: the text is fluent and plausible
ControlsQuote-grounded extraction and citation check (preventive); stratified human sampling (detective); adjuster sign-off before any denial (gate)
Owner and triggerClaims platform lead; re-score on model change or new document type

Three habits separate a defensible register from a checklist. Estimate likelihood from measurements on your own task distribution (the evaluation guidance says to mirror real-world distribution and include edge cases), not from intuition. Rank by impact first and detectability second: a high-impact, hard-to-detect, irreversible risk needs a gate, whereas monitoring alone suits low-impact, reversible ones. And decide explicitly whether each risk is mitigated, accepted, transferred or avoided, with the accepting owner named. Track a few key risk indicators (unsupported-claim rate on the sample, reviewer override rate, refusal rate, eval score on the pinned model) so drift shows up as a number before it shows up as an incident.

Key concept

Risk lives in the system, not the model: the same limitation is a nuisance in one workflow and a liability in another. Measure rates on your own distribution, classify controls as preventive, detective or corrective, prioritise by impact and detectability, and re-score whenever the model, data or scope changes.

Exam traps

Practice question

A multi-agent research pipeline (coordinator, search subagent, summarising subagent, report writer) produced a customer-facing report containing a precise market statistic that appears in no source. The summariser had presented it confidently, and the writer repeated it. Which control set best addresses the underlying failure mode?

  • A Set the sampling temperature to zero for every agent in the pipeline so that outputs become consistent and stop inventing facts, and record the setting alongside each run for later audits.

    Consistency is not correctness. A model can consistently produce the same unsupported number, and the sampling parameters are deprecated on the newest models, so this cannot be the control.

  • B Add a final editor agent that rewrites the report to sound more cautious, without giving it access to the sources, so that any unsupported claims are hedged before customers ever see them.

    Hedged wording does not remove a fabricated figure, and an editor without sources cannot verify anything. It only makes the false claim harder for a human to spot.

  • C Upgrade every agent to the newest model and remove the verification steps to reduce latency and cost, since newer models hallucinate less and the extra checks no longer add value.

    A newer model may lower the rate, but the docs say mitigations reduce rather than eliminate hallucination and that critical information must still be validated. Removing controls converts an improvement into unmanaged risk, and the upgrade itself needs regression testing.

  • D Require each handoff to carry supporting source quotes, validate every figure against them before the next agent uses it, cap iterations, and add a human checkpoint before external publication. Correct

    This attacks compounding errors at the point where they propagate: grounding and verification at each handoff (quote-and-cite technique), a stopping condition, and a human gate proportional to the consequence of publishing.

Build exercise: Build a risk register for an LLM workflow and measure one risk

Intermediate · 60 minutes

You'll practice:

  1. Pick a workflow (for example, an agent that summarises insurance claim documents, retrieves policy text and drafts a decision letter). Write half a page describing the users, the decisions the output influences, the tools, and the untrusted inputs.

    Risk is defined relative to consequence. Without the decisions and the untrusted inputs written down, every rating in the register is a guess.

    You should see: A context paragraph naming who acts on the output, what they can do with it, whether it reaches an external party, and which inputs come from outside your control.

    Hints 
    1. Who could be harmed if the output is wrong, and can the harm be undone?
    2. Write the context so that someone outside the team could rate impact from it: name the decision, the person affected and the reversibility.
    3. Example: claims adjusters use the summary to approve or deny claims under a set amount; a wrong denial harms the policyholder and is reversible only after a complaint; policy PDFs and claimant emails are untrusted inputs; the letter is sent externally.

  2. Enumerate at least one risk for each of the six categories (hallucination, injection, over-reliance, drift, non-determinism, cascading errors). Write each as cause, event, consequence.

    Forcing all six categories prevents the common blind spot of registering only hallucination. The cause-event-consequence form makes each risk something you can design a test for.

    You should see: Six or more rows, each with a statement that names a mechanism specific to your workflow rather than a generic phrase such as 'AI may be wrong'.

    Hints 
    1. For each category, ask what would have to be true in your workflow for it to matter.
    2. Use the template: Because [cause], the agent [event], leading to [consequence for a named person]. Avoid statements that could describe any system.
    3. Drift row: Because the model version used for the letter drafter is retired, the replacement phrases denial reasons differently, leading to letters that no longer match the approved template. Non-determinism row: Because the same claim can be summarised differently on re-run, the audit record cannot be regenerated from the input alone.

  3. Rate likelihood, impact and detectability qualitatively, assign controls (tag each preventive, detective or corrective and note its layer), name an owner, and add a review trigger. Sort the register so high-impact, low-detectability, irreversible risks are first.

    The ordering is the decision: it shows where a gate in code or a human approval is required and where monitoring suffices.

    You should see: A sorted table in which every top-ranked risk has at least one preventive control in a deterministic layer, and every row has a named owner and a trigger.

    Hints 
    1. Which of your risks would you only learn about from a customer complaint?
    2. Rank by impact, then by how hard the failure is to notice. If the top row has only a detective control, add a gate or a preventive control.
    3. Top row example: unsupported policy exclusion cited in a denial. Preventive: quote-grounded extraction with citation check; gate: adjuster approves every denial; detective: weekly stratified sample; owner: claims platform lead; trigger: model change, new policy form.

  4. Measure one likelihood: run the same extraction prompt three times per document over a set of at least 20 documents, and compute how often the runs disagree. Record the rate per document type.

    Best-of-N comparison is one of the hallucination-detection techniques in Anthropic's guide. Disagreement is a cheap proxy for unreliable spots, and per-type breakdown avoids hiding a weak segment inside an average.

    You should see: A table of disagreement rate by document type, with at least one type noticeably worse than the overall figure, feeding a likelihood rating in the register.

    Hints 
    1. If the three runs agree, does that prove the answer is right? What would you compare against to find out?
    2. Extract one field with a strict output format, collect the three answers per document and count documents where they differ. Then compare against ground truth for a sample, because agreement is not accuracy.
    3. import anthropic, collections

client = anthropic.Anthropic()

def extract_total(doc):
    r = client.messages.create(
        model='claude-sonnet-5',  # check the current models page for available IDs
        max_tokens=50,
        messages=[{'role': 'user', 'content': 'Extract the claim total as digits only. If absent reply NONE.\n<doc>' + doc + '</doc>'}],
    )
    return r.content[0].text.strip()

runs = [extract_total(doc) for _ in range(3)]
disagree = len(set(runs)) > 1
print(collections.Counter(runs), 'DISAGREE' if disagree else 'agree')

  5. Add drift controls: record the exact model identifier in configuration, schedule the eval set to re-run, define an alert threshold for your key risk indicators, and write a migration checklist that starts testing a replacement model well before any retirement date.

    Drift is the risk that produces no error message. Only a pinned identifier, a regression set and a rehearsed migration turn a silent change into a visible, planned one.

    You should see: A config file with a single explicit model ID, a scheduled eval job, numeric alert thresholds for at least three indicators, and a checklist with owners and dates.

    Hints 
    1. What would tell you this week, not next quarter, that behaviour has moved?
    2. Pick indicators you can compute automatically: eval score on the pinned model, unsupported-claim rate on the sampled outputs, reviewer override rate, refusal rate. Set a threshold and an action for each.
    3. Checklist: 1) read the deprecations page and note retirement dates; 2) run the full eval set on the candidate model; 3) compare per-segment results, not just the average; 4) rerun red-team fixtures; 5) update the register rows that name the model; 6) cut over behind a canary; 7) keep the previous identifier until the retirement date.

Sources