PPassTheCertExam

Study guides / CCAR-F / Domain 4

Prompt Engineering & Structured Output · Lesson 1 of 6

4.1 - System Prompts with Explicit Criteria

Design prompts with explicit criteria to improve precision and reduce false positives

The single biggest mistake in production prompt engineering is relying on vague instructions. "Be conservative." "Only report high-confidence findings." "Use your best judgement." None of these give the model an actionable decision boundary. They sound reasonable, which is exactly why the exam uses them as distractors.

The correct approach is explicit categorical criteria that define precisely what the model should flag and what it should skip. Compare these two system prompts for a CI/CD code review pipeline:

Wrong approach:

Review this code. Be conservative. Only report high-confidence findings.

Correct approach:

Flag comments only when claimed behaviour contradicts actual code behaviour.
Report bugs and security vulnerabilities.
Skip minor style preferences and local patterns.

The first gives the model no criteria to apply. "Conservative" means different things in different contexts, and "high-confidence" is a subjective threshold the model cannot calibrate. The second provides concrete categories: what to report (bugs, security), what to skip (style, local patterns), and a specific trigger for comment flags (claimed vs actual behaviour contradiction).

The False Positive Trust Problem

High false positive rates in one category destroy developer trust in all categories. The exam leans on this hard. If your "documentation mismatch" findings are wrong 40% of the time, developers stop reading your "security vulnerability" findings too, even when those run at 98% accuracy. Trust isn't category-specific. It bleeds across the whole output.

The fix feels backwards: temporarily disable the high false-positive categories while you rework their prompts. Trust in the categories that already work comes back straight away. Then you iterate on the broken category with concrete code examples, switching it back on only once precision improves.

You're not abandoning the category. You're putting system-wide trust ahead of category completeness.

Severity Calibration with Code Examples

Defining severity levels requires concrete code examples, not prose descriptions. Compare:

Prose description (insufficient):

Critical: Issues that could cause system failures or data loss
Minor: Issues that affect code readability but not functionality

Code example approach (correct):

Critical - Unsanitised user input in SQL query:
  query = f"SELECT * FROM users WHERE id = {user_input}"

Minor - Inconsistent variable naming:
  userName vs user_name in the same module

The prose description forces the model to interpret what "could cause system failures" means. The code example removes ambiguity entirely. When the model sees actual code patterns classified at each severity level, it produces consistent classification across invocations.

Key Concept

Explicit categorical criteria always outperform vague instructions. Define what to flag (bugs, security vulnerabilities) and what to skip (style preferences, local patterns) using concrete code examples for each severity level. Never rely on "be conservative" or confidence-based filtering.

Why Confidence-Based Filtering Fails

The exam frequently presents "only report high-confidence findings" as a tempting answer. It sounds like good engineering: filter by confidence, keep only the strong signals. But LLM self-reported confidence is poorly calibrated. The model is often sure about wrong findings and hesitant about right ones. Confidence scores earn their keep in routing (sending low-confidence findings to human review, as covered in Task Statement 4.6), but they're no substitute for explicit criteria that define what counts as a valid finding in the first place.

The hierarchy is: explicit criteria first, confidence-based routing second. Never skip the first step.

Exam traps

Practice question

Your CI/CD code review pipeline has a 40% false positive rate on 'documentation mismatch' findings, causing developers to ignore ALL review categories including accurate security findings. What is the most effective fix?

  • A Add "only report high-confidence documentation issues" to the system prompt so the model filters its own weaker findings

    Vague confidence instructions do not improve precision. The model has no concrete criteria for what high-confidence means in this context.

  • B Add a second model pass that re-examines each documentation finding and discards any it cannot verify before the report reaches developers

    A second pass without better criteria will have the same false positive problem. Fix the root cause - the criteria - before adding verification layers.

  • C Increase the model temperature to produce more varied review runs, then filter out findings that appear only once

    Temperature affects randomness, not precision. Higher temperature would likely increase false positives rather than reduce them.

  • D Temporarily disable the documentation mismatch category while refining its prompts with explicit criteria and code examples Correct

    This restores trust in all other categories immediately while you iterate on the problematic category with specific, concrete criteria. Trust recovery across all categories is the priority.

Build exercise: Build an Explicit Criteria Code Review Prompt

Intermediate · 45 minutes

You'll practice:

  1. Write a system prompt with vague instructions (be conservative, only flag important issues) and test it against 5 code snippets containing known bugs, security issues, and style nitpicks

    Establishing a baseline with vague instructions demonstrates the false positive problem the exam tests. You need empirical evidence that phrases like be conservative give the model no actionable decision boundary.

    You should see: Inconsistent classification across the 5 snippets: some style nitpicks flagged as critical, some genuine bugs missed or marked minor, and different results if you run the same snippets twice.

    Hints 
    1. Think about what be conservative actually means to a model with no domain context.
    2. Include at least one SQL injection vulnerability, one unused variable, and one naming convention inconsistency in your test set so you cover multiple severity levels.
    3. const systemPrompt = `Review this code. Be conservative. Only report high-confidence findings.`;

const testSnippets = [
  { code: "query = f\"SELECT * FROM users WHERE id = {user_input}\"", expectedSeverity: "critical" },
  { code: "const unused_var = 42;", expectedSeverity: "minor" },
  { code: "let userName = getUser(); let user_name = userName;", expectedSeverity: "style" }
];

  2. Rewrite the prompt with explicit categorical criteria: define exactly which issues to report (bugs, security vulnerabilities) and which to skip (style preferences, local patterns)

    Explicit categorical criteria are the correct approach tested on the exam. This step demonstrates that concrete categories eliminate the ambiguity that causes false positives.

    You should see: The rewritten prompt has clear categories: report bugs and security vulnerabilities, skip style preferences and local patterns, flag comments only when claimed behaviour contradicts actual code behaviour.

    Hints 
    1. Structure your criteria as a bulleted list with report and skip sections.
    2. Use the exam pattern: define trigger conditions (what to flag), exclusion conditions (what to skip), and the comment flag rule (claimed vs actual behaviour contradiction).
    3. const systemPrompt = `Flag comments only when claimed behaviour contradicts actual code behaviour.
Report: bugs, security vulnerabilities, logic errors.
Skip: minor style preferences, local naming patterns, formatting choices.
Do not flag unused imports unless they shadow a used import.`;

  3. Add concrete code examples for each severity level - critical, major, minor - showing actual code patterns, not prose descriptions

    The exam specifically tests that code examples outperform prose descriptions for severity calibration. Prose like issues that could cause system failures forces the model to interpret, while code examples remove ambiguity entirely.

    You should see: Your prompt now contains at least one code snippet per severity level, each showing the actual pattern that defines that severity, not a prose description of what that severity means.

    Hints 
    1. Each severity example should show a recognisable code anti-pattern, not just describe the concept.
    2. Include the pattern name alongside each code example so the model learns to identify the construct, not just match syntax.
    3. Add to your prompt:

Critical - Unsanitised user input in SQL query:
  query = f"SELECT * FROM users WHERE id = {user_input}"

Major - Missing null check before property access:
  const name = response.data.user.name; // no null guard

Minor - Inconsistent variable naming:
  userName vs user_name in the same module

  4. Compare false positive rates between the two versions on the same test set and document which approach produces more consistent classification

    Quantifying the improvement validates the explicit criteria approach and builds the evaluation skill the exam expects. You should be able to articulate why one approach outperforms the other with data, not intuition.

    You should see: A clear reduction in false positives with the explicit criteria version. The vague prompt should produce 30-50% inconsistency while the explicit criteria version should be below 15%. Classification should be stable across repeated runs.

    Hints 
    1. Run each prompt version 3 times against the same test set to check for consistency, not just accuracy.
    2. Track three metrics per run: true positives (correctly flagged bugs), false positives (style issues flagged as bugs), and false negatives (missed bugs). Calculate precision = TP / (TP + FP).
    3. const results = { vague: { tp: 0, fp: 0, fn: 0 }, explicit: { tp: 0, fp: 0, fn: 0 } };
// Run both prompts against each snippet
for (const snippet of testSnippets) {
  const vagueResult = await classify(vaguePrompt, snippet);
  const explicitResult = await classify(explicitPrompt, snippet);
  // Compare against expected severity
  updateMetrics(results.vague, vagueResult, snippet.expectedSeverity);
  updateMetrics(results.explicit, explicitResult, snippet.expectedSeverity);
}

  5. Temporarily disable any category with above 25% false positive rate and document the criteria refinements needed before re-enabling

    The trust recovery strategy is a key exam concept: high false positive rates in one category destroy developer trust in ALL categories. Disabling problematic categories restores system-wide trust while you iterate on their criteria.

    You should see: A document listing which categories exceed the 25% threshold, what specific criteria refinements are needed (e.g., add code examples for edge cases), and a re-enablement plan with target false positive rates.

    Hints 
    1. Focus on the trust bleed effect: explain how disabling one noisy category improves perceived accuracy of the remaining categories.
    2. For each disabled category, write 2-3 concrete code examples covering the edge cases that caused false positives. These become your refined criteria for re-enablement.
    3. const categoryMetrics = {
  documentation_mismatch: { fp_rate: 0.40, action: "disable" },
  security_vulnerability: { fp_rate: 0.05, action: "keep" },
  logic_error: { fp_rate: 0.12, action: "keep" }
};

// Refinement plan for disabled categories
const refinements = {
  documentation_mismatch: {
    issue: "Flagging outdated comments as mismatches when code is correct",
    fix: "Add examples distinguishing stale comments from genuine contradictions",
    targetFpRate: 0.15
  }
};

Sources