4.1 - System Prompts with Explicit Criteria

Build exercise: Rewrite a vague instruction with explicit criteria and measure the difference

Beginner · 30 minutes

You'll practice:

• Convert a vague goal statement into explicit, decidable criteria
• Encode a precision/recall trade-off explicitly for a borderline case
• Call the Messages API with a system prompt and compare consistency across repeated runs
• Recognise and correct a noisy, trust-damaging category in a multi-category prompt

1. Take a vague instruction (e.g. "flag risky support requests") and rewrite it naming at least three concrete criteria a human reviewer would actually check, including how to handle one specific borderline case (e.g. a customer joking about cancelling their account).

   Naming the borderline case explicitly is usually where the real ambiguity lived - general categories alone don't resolve it.

   You should see: A rewritten instruction specific enough that two different people applying it by hand would make the same call on the borderline case.

   Hints

   1. What would a human reviewer actually check line by line before deciding? Write that list down first, then turn it into the prompt.

   2. Structure it as report/skip categories plus one explicit rule for the borderline case, the same shape as the SQL-injection vs. style-nit example in the lesson.

   3. const criteria = `Flag: explicit cancellation requests, threats of chargeback, repeated unresolved complaints.\nDo not flag: general product feedback, jokes or hyperbole (\"I'll cancel if this happens again\" said in a positive-toned message) unless paired with an explicit action request.\nWhen genuinely unclear, flag for human review rather than silently skip.`;

2. Send the vague version and the explicit-criteria version as separate system prompts against the same 5 borderline test messages using the Messages API, and log each response.

   Seeing both versions run against identical input is the only way to observe the actual behaviour difference - reasoning about it in the abstract hides how much the vague version drifts.

   You should see: Two sets of five classifications. The vague-prompt set should show at least one inconsistency (a case classified differently from a near-identical case); the explicit-criteria set should be uniform.

   Hints

   1. The system prompt goes in the top-level `system` parameter of messages.create, not as a user turn - keep the borderline test messages identical across both runs so it's a fair comparison.

   2. Loop over the same array of 5 message strings for both system prompts and store (systemVersion, input, output) tuples so you can diff them side by side afterward.

   3. const testMessages = [/* 5 borderline strings */];
      for (const sys of [vagueSystemPrompt, explicitSystemPrompt]) {
        for (const msg of testMessages) {
          const res = await client.messages.create({
            model: "claude-sonnet-5",
            max_tokens: 200,
            system: sys,
            messages: [{ role: "user", content: msg }],
          });
          results.push({ sys: sys === vagueSystemPrompt ? "vague" : "explicit", msg, out: res.content[0].text });
        }
      }

3. Run the explicit-criteria version 3 times against the same 5 messages to check for run-to-run consistency, not just single-run correctness.

   A prompt can look fine on one run and still drift across repeated calls if the criteria aren't specific enough - consistency across repeats is the real test the exam cares about.

   You should see: The same classification for each message across all 3 runs. Any message that flips between runs still has residual ambiguity worth tightening further.

   Hints

   1. What would you compare to detect drift - the raw text, or a normalised label extracted from it?

   2. Extract a single label (e.g. "flag" / "no_flag") from each response with a small parsing step, then group by message and check that all 3 runs agree.

   3. const runs = await Promise.all([1, 2, 3].map(() => classifyAll(explicitSystemPrompt, testMessages)));
      for (let i = 0; i < testMessages.length; i++) {
        const labels = runs.map(r => r[i].label);
        console.log(testMessages[i], labels, new Set(labels).size === 1 ? "CONSISTENT" : "DRIFTED");
      }

4. Add a second category to the system prompt (e.g. "tone issues" alongside "cancellation risk") and deliberately leave its criteria vague, then observe whether reviewers (or your own spot-check) start distrusting the well-defined category too.

   This recreates the false-positive trust problem in miniature - a noisy category sitting next to a solid one, in the same prompt, is the scenario the exam presents.

   You should see: The vague "tone issues" category produces inconsistent or clearly wrong flags on your test set, while "cancellation risk" stays accurate - demonstrating the two categories are independently reliable even though they'd read as equally untrustworthy to a downstream reviewer skimming both outputs together.

   Hints

   1. You don't need real reviewer feedback for this - just measure the false-positive rate of each category separately on your test set.

   2. Tag your test set with the expected category outcome ahead of time, then compute precision per category the same way you would for the code-review scenario in the lesson.

   3. const categories = { cancellation_risk: { tp: 0, fp: 0 }, tone_issues: { tp: 0, fp: 0 } };
      // after tagging outputs against expected labels:
      const precision = c => c.tp / (c.tp + c.fp || 1);
      console.log("cancellation_risk precision:", precision(categories.cancellation_risk));
      console.log("tone_issues precision:", precision(categories.tone_issues));