Study guides / CCAR-F / Domain 2

Tool Design & MCP Integration · Lesson 1 of 5

2.1 - Tool Interface Design

Designing effective tool interfaces with clear descriptions and boundaries for reliable LLM tool selection

Tool descriptions are the PRIMARY mechanism LLMs use for tool selection. Not supplementary metadata. Not an afterthought. The mechanism. When a model receives a set of tools, it reads the descriptions to decide which one to call - and if those descriptions are minimal, something like "Retrieves customer information", it has no way to tell apart tools that serve overlapping purposes.

What Makes a Good Tool Description

A production-grade tool description includes five elements:

  1. What the tool does - its primary purpose, stated unambiguously
  2. What inputs it expects - data types, formats, constraints, and required versus optional fields
  3. Example queries it handles well - concrete use cases that anchor the model's understanding
  4. Edge cases and limitations - what the tool does NOT do, and what happens when inputs fall outside expected ranges
  5. Explicit boundaries - when to use THIS tool versus similar tools in the same toolkit

Here is the difference between a minimal and a production-grade description:

Minimal (causes misrouting):

get_customer: "Retrieves customer information" lookup_order: "Retrieves order details"

Production-grade (reliable selection):

get_customer: "Looks up a customer account by email address, phone number, or customer ID. Returns customer profile (name, contact details, account status, loyalty tier). Use this when you need to verify who the customer is. Do NOT use for order-specific queries - use lookup_order for those."

lookup_order: "Retrieves order details by order number (format: #NNNNN) or tracking ID. Returns order status, items, shipping details, and refund eligibility. Use this when a customer asks about a specific order. Do NOT use for customer identity verification - use get_customer for that."

The second version gives the model explicit disambiguation. It knows which identifiers each tool accepts, what each returns, and crucially, when NOT to use each tool.

The Misrouting Problem

Two tools with overlapping or near-identical descriptions cause selection confusion. The exam's Q2 presents exactly this scenario: get_customer and lookup_order with minimal descriptions, causing the agent to route "check my order #12345" to the wrong tool.

The exam tests whether you can spot the correct fix. Four plausible options, three of them wrong:

The exam consistently favours low-effort, high-leverage fixes. Better descriptions before routing classifiers. Scoped access before full access. Community servers before custom builds.

Tool Splitting

Generic tools with broad responsibilities create ambiguity. The fix: split them into purpose-specific tools with defined input/output contracts.

Before splitting:

analyze_document: "Analyses a document and returns results"

After splitting:

extract_data_points: "Extracts structured data fields (dates, amounts, names) from a document"

summarize_content: "Produces a concise summary of a document's key arguments and conclusions"

verify_claim_against_source: "Checks whether a specific claim is supported by the source document, returning supporting/contradicting evidence"

Each resulting tool does one narrow, clearly described job. The model can pick the right one based on what the user actually needs.

Tool Renaming for Clarity

When two tools have confusingly similar names, renaming fixes the overlap at the interface level. Rename analyze_content to extract_web_results, give it a web-specific description, and the tool's purpose becomes unambiguous - without touching its implementation.

System Prompt Interactions

Keyword-sensitive instructions in system prompts can create unintended tool associations that override well-written descriptions. If your system prompt says "always check customer details before proceeding", the model may route any customer-related query to get_customer no matter what the descriptions say.

So after updating tool descriptions, reread your system prompt for conflicts. It's a subtle failure mode, and the exam tests it.

Key Concept

Tool descriptions are the primary mechanism LLMs use for tool selection. When misrouting is caused by weak descriptions, improving them is the first fix - not few-shot examples, routing classifiers, or tool consolidation.

Read the condition on that, because the exam tests both halves. Descriptions are the fix when the agent has a workable number of tools and simply cannot tell two of them apart. They are not the fix when the toolkit itself is the problem: past roughly 4-5 tools per agent, selection degrades on decision complexity alone, and rewriting 22 descriptions leaves that untouched. Diagnose which one you are looking at before reaching for a remedy. Task Statement 2.3 covers the overload threshold and what to do instead.

Exam traps

Practice question

Production logs show an agent frequently calls get_customer when users ask about orders (e.g. 'check my order #12345'), instead of calling lookup_order. Both tools have minimal descriptions ('Retrieves customer information' / 'Retrieves order details') and accept similar identifier formats. What is the most effective first step to improve tool selection reliability?

  • A Add 5-8 few-shot examples to the system prompt demonstrating correct tool selection patterns for order-related queries.

    Few-shot examples add token overhead without fixing the underlying issue. The root cause is that descriptions do not differentiate the tools - fix the descriptions first.

  • B Expand each tool description to include input formats, example queries, edge cases, and boundaries explaining when to use it versus similar tools. Correct

    Tool descriptions are the primary mechanism LLMs use for tool selection. Expanding them is the lowest-effort, highest-leverage fix that directly addresses the root cause of misrouting.

  • C Implement a routing layer that parses user input before each turn and pre-selects the appropriate tool based on detected keywords.

    A routing layer is over-engineered as a first step. It bypasses the LLM's natural language understanding and adds unnecessary infrastructure complexity.

  • D Consolidate both tools into a single lookup_entity tool that accepts any identifier and internally determines which backend to query.

    Consolidation is a valid architectural choice but requires significantly more effort than expanding descriptions. The exam favours proportionate first steps.

Build exercise: Design Tool Descriptions That Eliminate Misrouting

Beginner · 30 minutes

You'll practice:

  1. Create two MCP tools with intentionally ambiguous descriptions (e.g. get_customer: Retrieves customer information and lookup_order: Retrieves order details)

    Reproducing a misrouting scenario first-hand builds intuition for why minimal descriptions fail. The exam tests your ability to identify ambiguous descriptions as the root cause of tool selection errors.

    You should see: Two tool definitions registered with your MCP server, each having a single-sentence description that does not mention input formats, example queries, or boundaries.

    Hints
    1. Think about what information is missing - what inputs does each tool accept? When should one be used instead of the other?
    2. Define each tool using the MCP SDK inputSchema with a string parameter for the identifier. Keep descriptions to one generic sentence each.
    3. import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";
      const server = new McpServer({ name: "customer-tools", version: "1.0.0" });
      server.tool("get_customer", "Retrieves customer information", { identifier: { type: "string" } }, async ({ identifier }) => {
        return { content: [{ type: "text", text: `Customer data for ${identifier}` }] };
      });
      server.tool("lookup_order", "Retrieves order details", { identifier: { type: "string" } }, async ({ identifier }) => {
        return { content: [{ type: "text", text: `Order data for ${identifier}` }] };
      });
  2. Test with 10 queries covering different user intents and log which tool the model selects for each

    Quantifying selection accuracy before and after description changes gives you concrete evidence of the impact. The exam expects you to know that description quality directly affects selection reliability.

    You should see: A log showing at least 2-3 misrouted queries where the model selected get_customer for order-related queries or vice versa, demonstrating the ambiguity problem.

    Hints
    1. Include queries that mention order numbers, customer emails, tracking IDs, and ambiguous phrases like "check my account" to cover edge cases.
    2. Create a test harness that sends each query to the Claude API with both tools available and tool_choice set to auto, then logs which tool was called.
    3. const queries = [
        "What is the status of order #12345?",
        "Look up customer john@example.com",
        "Check my order tracking",
        "Find the account for phone 555-0123",
        "Where is my package?",
        "Is order #67890 eligible for a refund?",
        "What loyalty tier is this customer?",
        "I need details on order #11111",
        "Verify the customer account status",
        "When will order #99999 arrive?"
      ];
      for (const query of queries) {
        const response = await client.messages.create({
          model: "claude-sonnet-5",
          max_tokens: 1024,
          tools: toolDefinitions,
          messages: [{ role: "user", content: query }]
        });
        const toolUse = response.content.find(b => b.type === "tool_use");
        console.log(`Query: ${query} => Tool: ${toolUse?.name}`);
      }
  3. Rewrite both descriptions to include: purpose, expected inputs with formats, example queries, edge cases, and explicit boundaries against the other tool

    This is the core exam skill - the lowest-effort, highest-leverage fix for misrouting. Production-grade descriptions include all five elements: purpose, inputs, examples, edge cases, and boundaries.

    You should see: Each tool description is 3-5 sentences long, explicitly states accepted identifier formats, gives example queries, and includes a boundary statement like "Do NOT use for order-specific queries - use lookup_order for those."

    Hints
    1. For each tool, answer five questions: What does it do? What inputs does it accept? What queries suit it? What does it NOT handle? When should the other tool be used instead?
    2. Include specific format examples (e.g. email addresses, phone numbers, order numbers like #NNNNN) and explicitly state the return data shape for each tool.
    3. server.tool("get_customer", "Looks up a customer account by email address, phone number, or customer ID. Returns customer profile (name, contact details, account status, loyalty tier). Use this when you need to verify who the customer is or check account details. Do NOT use for order-specific queries - use lookup_order for those.", { identifier: { type: "string", description: "Customer email, phone, or ID" } }, async ({ identifier }) => {
        return { content: [{ type: "text", text: `Customer profile for ${identifier}` }] };
      });
  4. Re-run the same 10 queries and compare selection accuracy before and after

    Measuring improvement validates that description quality is the root cause. The exam expects you to understand that better descriptions produce measurably better selection without any architectural changes.

    You should see: Selection accuracy improves to 9/10 or 10/10 correct, with previously misrouted queries now hitting the correct tool. A clear before/after comparison showing the improvement.

    Hints
    1. Run the exact same 10 queries and compare results side by side. Pay special attention to the queries that were misrouted before.
    2. Create a simple comparison table logging query, expected tool, tool selected before, and tool selected after. Calculate accuracy percentage for both runs.
    3. let correct = 0;
      const expected = ["lookup_order", "get_customer", "lookup_order", "get_customer", "lookup_order", "lookup_order", "get_customer", "lookup_order", "get_customer", "lookup_order"];
      for (let i = 0; i < queries.length; i++) {
        const response = await client.messages.create({ model: "claude-sonnet-5", max_tokens: 1024, tools: updatedToolDefinitions, messages: [{ role: "user", content: queries[i] }] });
        const toolUse = response.content.find(b => b.type === "tool_use");
        const match = toolUse?.name === expected[i];
        if (match) correct++;
        console.log(`${match ? "CORRECT" : "WRONG"} | ${queries[i]} => ${toolUse?.name} (expected: ${expected[i]})`);
      }
      console.log(`Accuracy: ${correct}/${queries.length}`);
  5. Review your system prompt for keyword-sensitive instructions that could override the improved descriptions

    System prompt conflicts are a subtle failure mode the exam tests. Keywords like "always check customer details" can create unintended tool associations that override even well-written descriptions.

    You should see: A list of any keyword-sensitive phrases in your system prompt that could trigger incorrect tool associations, along with rewritten versions that avoid the conflict.

    Hints
    1. Search for words like "customer", "order", "check", "verify", "look up" in your system prompt - these could trigger unintended associations with specific tools.
    2. Compare tool selection results with and without the system prompt. If accuracy drops with the prompt, identify which phrases are causing the interference.
    3. // Test with a system prompt that contains a conflicting instruction
      const conflictingPrompt = "Always check customer details before proceeding with any request.";
      const response = await client.messages.create({
        model: "claude-sonnet-5",
        max_tokens: 1024,
        system: conflictingPrompt,
        tools: updatedToolDefinitions,
        messages: [{ role: "user", content: "What is the status of order #12345?" }]
      });
      // If get_customer is selected instead of lookup_order, the system prompt is interfering

Sources