5.2 - Escalation & Ambiguity Resolution

Build exercise: Build an Escalation Decision Engine

Intermediate · 40 minutes

You'll practice:

• Implement the three valid escalation triggers: explicit human request, policy exceptions/gaps, and inability to progress
• Identify and avoid the two unreliable triggers: sentiment-based escalation and self-reported confidence scores
• Handle the frustration nuance: frustrated customer with resolvable issue vs explicit human request
• Design ambiguous customer matching that requests additional identifiers rather than selecting heuristically
• Add explicit escalation criteria with few-shot examples to system prompts as the proportionate first response

1. Create a system prompt with explicit escalation criteria covering all three valid triggers: explicit human request, policy exceptions/gaps, and inability to make progress

   Explicit escalation criteria in the system prompt are the proportionate first response before adding infrastructure like classifier models or sentiment analysis. The exam tests that prompt optimisation should always precede architectural changes for escalation calibration.

   You should see: A system prompt with three clearly defined escalation triggers, each with a description and decision rule. The prompt should also explicitly list the two anti-patterns (sentiment-based and confidence-based escalation) as things to avoid.

   Hints

   1. Structure the criteria as a numbered list with trigger, description, and action for each of the three valid triggers.

   2. Include the distinction between policy gaps (policy is silent on this situation) and policy violations (the policy says no). Gaps require escalation; violations have documented answers.

   3. const systemPrompt = `## Escalation Criteria\n\n` +
        `1. EXPLICIT HUMAN REQUEST: If the customer says they want a human, escalate IMMEDIATELY. Do not attempt to resolve first.\n` +
        `2. POLICY GAP: If the request falls outside documented policy (not a violation, but a gap where policy is silent), escalate for human judgement.\n` +
        `3. INABILITY TO PROGRESS: If you have attempted resolution and cannot advance (tool errors, missing system access, engineering intervention needed), escalate with context.\n\n` +
        `DO NOT escalate based on: customer frustration level, your own confidence score, or sentiment analysis.`;

2. Add few-shot examples showing: immediate escalation for explicit human request, autonomous resolution for a frustrated customer with a straightforward issue, and escalation for a policy gap

   Few-shot examples demonstrating when to escalate versus when to resolve autonomously directly address unclear decision boundaries. This is the exact technique the exam identifies as the correct improvement for a support agent with poor first-contact resolution rates.

   You should see: Three examples in the system prompt, each showing a different scenario with the correct decision and reasoning. The frustrated-but-resolvable example should show the agent acknowledging frustration and offering the resolution directly.

   Hints

   1. Each example should show the customer message, the agent reasoning, and the action taken. The reasoning is critical for teaching generalisation.

   2. The key distinction to demonstrate: frustrated customer with simple issue = resolve it, customer who explicitly says I want a human = escalate immediately.

   3. const examples = `\n## Examples\n\n` +
        `Customer: I want to speak to a real person right now.\n` +
        `Action: ESCALATE IMMEDIATELY. Customer explicitly requested a human. Do not investigate or offer to help first.\n\n` +
        `Customer: This is ridiculous! My package is a week late and nobody seems to care!\n` +
        `Action: RESOLVE. Issue is straightforward (late delivery). Acknowledge frustration, apologise, offer reshipment or compensation.\n` +
        `Response: I understand this is frustrating. I can see your package is delayed. Let me arrange a replacement shipment right now.\n\n` +
        `Customer: I purchased this from a competitor but want to return it at your store. Your website does not say anything about this.\n` +
        `Action: ESCALATE. Policy gap - competitor returns are not covered in documented policy. Requires human judgement on whether to make an exception.`;

3. Implement ambiguous customer matching logic that requests additional identifiers (email, phone, order number) instead of selecting heuristically

   Selecting from ambiguous matches using heuristics (most recent, most active) risks privacy violations and incorrect actions. The exam tests that the only safe response to multiple customer matches is to ask for additional identifiers to disambiguate.

   You should see: A matching function that detects when multiple records are returned and immediately asks for disambiguation rather than applying any selection heuristic. The disambiguation request should suggest specific identifier types.

   Hints

   1. Never select the most recent or most active record. Any heuristic-based selection risks exposing one customer data to another.

   2. The disambiguation prompt should offer multiple identifier options: email, phone number, order number, or postcode. Let the customer choose which to provide.

   3. function handleCustomerLookup(results) {
        if (results.length === 0) {
          return { action: "not_found", message: "No matching customer records found." };
        }
        if (results.length === 1) {
          return { action: "matched", customer: results[0] };
        }
        // Multiple matches - NEVER select heuristically
        return {
          action: "disambiguate",
          message: "I found multiple accounts matching that name. " +
            "Could you provide one of the following to help me find the right account?\n" +
            "- Email address\n- Phone number\n- Order number\n- Postcode",
          matchCount: results.length
        };
      }

4. Test with four scenarios: frustrated customer with simple issue, calm customer requesting policy exception, customer explicitly requesting a human, and ambiguous customer match

   These four scenarios cover all critical decision boundaries the exam tests: the frustration nuance, policy gap versus violation distinction, absolute rule for explicit human requests, and privacy-safe disambiguation.

   You should see: Correct handling of all four scenarios: resolution offered for the frustrated customer, escalation for the policy gap, immediate escalation for the explicit human request (no investigation first), and disambiguation request for the ambiguous match.

   Hints

   1. The critical check for scenario 3 is that the agent escalates IMMEDIATELY without any investigation or let me try first response.

   2. Verify the frustrated customer scenario does NOT trigger escalation. Frustration with a resolvable issue should be resolved, not escalated.

   3. const testScenarios = [
        { input: "This is SO frustrating! My order arrived broken!", expected: "resolve", reason: "Straightforward damage replacement" },
        { input: "I want you to match the price I saw at a competitor.", expected: "escalate", reason: "Policy gap - competitor matching not in policy" },
        { input: "I want to talk to a real human being.", expected: "escalate_immediate", reason: "Explicit human request - no investigation" },
        { input: "Look up my account, name is John Smith.", expected: "disambiguate", reason: "Multiple matches, ask for additional identifier" }
      ];
      
      for (const scenario of testScenarios) {
        const result = await processScenario(scenario.input);
        console.assert(result.action === scenario.expected, `Failed: ${scenario.reason}`);
      }

5. Verify the agent never attempts investigation before honouring an explicit human request and never selects from ambiguous matches using heuristics

   These are the two absolute rules the exam tests with no exceptions. Any attempt to investigate before escalating on an explicit human request, or any heuristic selection from ambiguous matches, is a critical failure that would cost marks on the exam.

   You should see: For explicit human requests: the escalation happens in the very first response with zero investigation steps. For ambiguous matches: the response always asks for additional identifiers, never selects a record. Both rules should hold across multiple phrasings and edge cases.

   Hints

   1. Test multiple phrasings of explicit human requests: transfer me to a human, I want a person, let me speak to someone real, get me a manager.

   2. For ambiguous matches, test that even when one match seems more likely (same city, recent activity), the system still asks for disambiguation rather than selecting.

   3. // Test absolute rules with varied phrasings
      const humanRequests = [
        "Transfer me to a human.",
        "I want to speak to a real person.",
        "Get me a manager.",
        "Let me talk to someone real, not a bot."
      ];
      
      for (const request of humanRequests) {
        const result = await processScenario(request);
        console.assert(result.action === "escalate_immediate", "Must escalate immediately");
        console.assert(result.investigationSteps === 0, "Zero investigation before escalation");
      }
      
      // Test ambiguous match with tempting heuristic
      const ambiguousResults = [
        { name: "John Smith", lastActive: "2024-03-14", city: "London" },
        { name: "John Smith", lastActive: "2023-01-02", city: "Manchester" }
      ];
      const matchResult = handleCustomerLookup(ambiguousResults);
      console.assert(matchResult.action === "disambiguate", "Never select heuristically");