PPassTheCertExam

Study guides / CCAR-P / Domain 1

Integration · Lesson 8 of 8

1.8 - Progressive Discovery vs a Monolithic Context Strategy

Weigh loading everything up front against discovering tools, skills, files and knowledge just in time, and choose a hybrid supported by measurements of cost, accuracy, latency and discovery failures.

Every agent design answers a question that is easy to skip: what is in the context on the first token, and what is fetched later? A monolithic strategy loads everything up front: all tool definitions, all instructions, the whole knowledge base. A progressive discovery strategy loads a lightweight index first and pulls detail in only when a task needs it. Neither is universally better. The exam asks you to weigh cost, accuracy, latency and failure modes for a described workload and to notice when a candidate answer quietly assumes the wrong side of the trade.

Where the choice appears

LayerMonolithicProgressive mechanism
ToolsEvery definition in every requestTool search with defer_loading; in MCP hosts, a search_tools meta-tool and connecting servers on demand
Instructions and expertiseOne very large system promptAgent Skills: name and description always loaded (about 100 tokens per skill), the SKILL.md body when triggered, bundled files and scripts only when used
KnowledgeThe whole corpus in the promptRetrieval (Lessons 1.5 and 1.6)
Code and dataPre-loaded files and large tool resultsJust-in-time exploration with tools such as grep and glob; code execution that filters data before it reaches the model

Anthropic's own numbers show the scale. Its tool search docs put a five-server setup at roughly 55k tokens of definitions and say selection accuracy degrades beyond roughly 30–50 tools. Its engineering write-up on code execution with MCP gives an illustrative case where moving from loading all tool definitions to discovering them on demand cut usage from about 150,000 tokens to about 2,000. Treat these as illustrations of magnitude, not guarantees.

The trade-offs, both directions

Monolithic is strong when the context is small. It is the simplest design, the model always sees everything so nothing can go undiscovered, there is one round trip, an unchanging prefix caches well, and debugging is easy because you know exactly what the model saw. It fails as content grows: tokens and time to first token are paid on every request, selection and focus degrade, and you approach context limits.

Progressive is strong when the context is large and each task needs a small part of it. It keeps the window focused and scales far beyond what fits (tool search supports up to 10,000 deferred tools per request). Its costs are real:

Key concept: progressive discovery moves the cost, and the failure mode

It shifts cost from every request to only the requests that need a capability, and it replaces "too much context" with "the model may not find it". A good answer names both, and says what you measure to know which risk you actually have.

Tool search and skills in practice

Server-side tool search offers two variants: tool_search_tool_regex_20251119 (Claude writes Python regex patterns, up to 200 characters) and tool_search_tool_bm25_20251119 (natural-language queries, up to 500 characters). You include the search tool, mark the tools to hold back with defer_loading: true, and at least one tool must stay non-deferred. Searches return matching tools as tool_reference blocks, five by default, which the API expands into full definitions. Two details are frequently missed. First, defer_loading controls what enters the context, not what you send: you still send every definition in tools on each request. Second, a deferred tool cannot also carry cache_control. For MCP connector toolsets you set defer_loading in default_config or per tool in configs. You can also implement your own search, for example with embeddings, by returning tool_reference blocks from a custom tool.

The docs give guidance on when it pays: 10 or more tools, definitions above about 10k tokens, degrading selection, or several MCP servers; standard loading is the better fit for fewer than 10 tools or when every tool is used on every request. The MCP client guidance suggests a threshold expressed as a share of the context window (for example 1–5%) at which a host switches to progressive discovery. Claude Code enables tool search for MCP tools by default in recent versions.

Skills apply the same idea to instructions: a Skill's description decides whether it triggers, so it must say both what it does and when to use it. Scripts run through bash and only their output enters the context. Only use Skills from sources you trust, as the docs warn that a malicious Skill can direct tool use and data exposure.

Choosing and combining

The usual answer is a hybrid: keep a small always-on core and make the long tail discoverable. Claude Code is the model case in Anthropic's write-up: project instruction files are loaded up front while files are explored just in time. For long exploration sessions, Anthropic lists compaction, structured note-taking into external memory and sub-agents with clean contexts that return distilled summaries as complementary techniques (Lesson 6.4). To decide, measure rather than guess: what share of the context do definitions and instructions consume, how often is each item actually used, what is the selection accuracy, and what does an extra discovery round trip add to latency?

Common exam distractor

"A larger context window makes discovery unnecessary" ignores per-request cost, latency and selection accuracy. "Deferred loading shrinks the request payload" is false: definitions are still sent, only the context is smaller. "Progressive discovery is always better" ignores the extra round trip and discovery misses for small toolsets. "Add or remove tools freely each turn" forgets prompt cache invalidation.

Exam traps

Practice question

A platform team's coding assistant loads all 40 team runbooks into one roughly 90k-token system prompt on every request. Any given task needs two or three runbooks, the runbooks change weekly, and the team reports slow first responses and occasional answers that apply the wrong runbook. A colleague proposes relying on prompt caching to make the large prompt cheap. Which approach best addresses the problem?

  • A Keep a small always-on core and expose each runbook as a discoverable unit (for example a Skill) with a precise 'what it covers and when to use it' description, then measure discovery misses and latency Correct

    It matches a large body of content where each task needs a small part, keeps the window focused, and confines weekly edits to individual units. Measuring discovery misses accounts for the new failure mode, and the clear descriptions are what make triggering reliable.

  • B Keep the monolithic prompt, mark it for prompt caching, order the runbooks by how often they are used and add a line telling the model to pick the runbook that matches the task, since cached tokens are cheaper to re-read

    Caching can reduce the cost and latency of re-reading an unchanged prefix, but weekly edits invalidate it and it does not change what the model must sift through, so wrong-runbook errors remain.

  • C Move to a model with a larger context window so that all 40 runbooks always fit comfortably, and keep the single system prompt as it is today

    The 90k tokens already fit; the reported problems are latency and misapplied runbooks, which a larger window does not fix and may worsen.

  • D Delete the half of the runbooks that are used least often to shrink the prompt and cut the cost of every request

    It discards knowledge users need instead of changing how it is loaded, and it does not solve the selection problem for the remaining runbooks.

Build exercise: Convert a monolithic tool set to deferred loading and measure the trade-off

Advanced · 100 minutes

You'll practice:

  1. Create or reuse an agent with at least 40 tools (synthetic tools are fine) and 20 tasks that each have one expected tool. Record the baseline: definition tokens (token counting) and how often the agent picks the expected tool.

    Without a baseline you cannot say whether deferral helped, and you need it to decide if the toolset is even large enough to justify it.

    You should see: Two numbers: input tokens attributable to tools, and selection accuracy on the 20 tasks.

    Hints 
    1. Which three tools are used most often across your tasks?
    2. Count tokens with and without the tools array using the token counting endpoint, and run each task with all tools loaded, logging which tool_use block appears first.
    3. baseline_tokens = client.messages.count_tokens(model=MODEL, tools=all_tools, messages=[{'role':'user','content':'ping'}]).input_tokens
For accuracy: run each task, take the first block with type 'tool_use', and compare its name to the expected tool.

  2. Enable tool search: add the BM25 search tool, mark every tool except your five most used with defer_loading true, and re-run the tasks. Record usage.input_tokens and accuracy.

    This is the core mechanism and the trade-off measurement: fewer context tokens, possibly different accuracy, extra searches.

    You should see: A smaller input token count on the first request, and accuracy compared with the baseline.

    Hints 
    1. Which tool must stay non-deferred, and what does the API do if all tools are deferred?
    2. Keep the hot five undeferred; the search tool itself must not be deferred. You still send every definition in the tools array.
    3. tools = [{'type':'tool_search_tool_bm25_20251119','name':'tool_search_tool_bm25'}] + [t if t['name'] in HOT else {**t, 'defer_loading': True} for t in all_tools]
resp = client.messages.create(model=MODEL, max_tokens=1024, tools=tools, messages=[{'role':'user','content':task}])
print(resp.usage.input_tokens)
Use a model from the tool search compatibility list.

  3. Log the searches. For each task record the search queries Claude issued, which tools were returned, and whether the expected tool was found. Compute the discovery miss rate.

    Discovery misses are the failure mode progressive loading introduces, and only logs let you see them.

    You should see: A table of task, query, tools returned, expected tool found (yes/no), plus a single miss-rate figure.

    Hints 
    1. For each miss, did the query fail to match the tool's name and description, or did the model not search at all?
    2. Look at server_tool_use blocks in the response content for the query, and at tool_search_tool_result for the tool references.
    3. queries = [b.input for b in resp.content if b.type == 'server_tool_use']
A miss is a task whose expected tool never appears in any search result, or where the model answers without searching.

  4. Improve discoverability without changing tool behaviour: add consistent name prefixes, keywords that match how users describe tasks, and a system prompt line listing the available tool categories. Re-measure misses, accuracy and tokens.

    This tests the documented mitigations and shows that discovery quality is engineered, not automatic.

    You should see: A lower miss rate and stable or better accuracy, with a before-and-after table.

    Hints 
    1. Which of the three changes moved the miss rate most?
    2. Change one thing at a time and re-run the same 20 tasks so the effect is attributable.
    3. System prompt line example: 'You can search for tools to work with GitHub, Slack, and Jira.' Rename search to github_search_issues; add words users actually use, such as 'ticket' and 'bug', to the description.

  5. Write the decision: for your toolset, at what size or context share do you switch from standard loading to deferred loading, what stays always-on, and what would make you switch back? Include latency evidence.

    The exam wants a justified threshold and a recognition of the cost of discovery, not a blanket preference.

    You should see: A short design note containing the measured context share, the accuracy and latency comparison, the always-on core, and switch-back conditions.

    Hints 
    1. If your toolset were only 8 tools, what would your measurements likely show?
    2. Compare total latency including the search round trip. If definitions are a small share of the context and every tool is used often, standard loading likely wins.
    3. Example: switch when definitions exceed about 5% of the context window or the tool count passes about 10, keep the top five tools and the search tool always loaded, revert to standard loading if the miss rate stays above the accuracy gain or the added round trip breaks the latency budget.

Sources