Study guides / CCAR-F / Domain 4

Prompt Engineering & Structured Output · Lesson 5 of 6

4.5 - Batch Processing Strategies

Design efficient batch processing strategies

The Message Batches API is a cost optimisation tool with hard constraints that the exam tests directly. Understanding when to use it - and when not to - is the core of this task statement.

Message Batches API: The Facts

The constraints are fixed, and you have to design around them:

The Matching Rule

This is the single most tested concept from this task statement:

Synchronous API: For blocking workflows where someone or something is waiting for the result. Pre-merge checks in CI/CD, real-time code review feedback, any workflow where developers are blocked pending completion.

Batch API: For latency-tolerant workflows where results are consumed later. Overnight technical debt reports, weekly code audit summaries, nightly test generation runs, batch document extraction.

The exam specifically presents a scenario (Question 11 in the sample questions) where a manager proposes switching everything to batch processing for the cost savings. The correct answer keeps blocking workflows synchronous and only moves latency-tolerant workflows to batch.

// Synchronous - developer is waiting for this
const preMergeReview = await client.messages.create({
  model: "claude-sonnet-5",
  max_tokens: 4096,
  messages: [{ role: "user", content: prDiffContent }]
});

// Batch - results consumed tomorrow morning
const batchRequest = await client.batches.create({
  requests: technicalDebtDocuments.map((doc, i) => ({
    custom_id: `debt-report-${i}`,
    params: {
      model: "claude-sonnet-5",
      max_tokens: 4096,
      messages: [{ role: "user", content: doc }]
    }
  }))
});

SLA Calculation

When designing batch processing schedules, you must account for the 24-hour maximum processing window. If your organisation requires a 30-hour SLA for a report:

The exam may present a scheduling question where you need to work backwards from the SLA to determine submission frequency.

Batch Failure Handling

Not all documents in a batch succeed. The correct failure handling pattern has three steps:

1. Identify failures by custom_id. Each request has a unique identifier. Parse the batch results to find which custom_id values failed.

2. Resubmit only failures with modifications. Do not resubmit the entire batch. Common modifications include:

3. Refine prompts on a sample set BEFORE batch processing. This is the proactive step that maximises first-pass success and reduces resubmission costs. Test your prompts against a representative sample (5-10 documents covering the range of formats and edge cases) before processing the full batch.

// Parse batch results and identify failures
const results = await client.batches.results(batchId);
const failures = results.filter(r => r.result.type === "errored");
const failedIds = failures.map(f => f.custom_id);

// Resubmit only failures with modifications
const retryRequests = failedIds.map(id => {
  const originalDoc = documentsById[id];
  return {
    custom_id: `${id}-retry-1`,
    params: {
      model: "claude-sonnet-5",
      max_tokens: 8192,  // increased for oversized docs
      messages: [{
        role: "user",
        content: chunkIfNeeded(originalDoc)
      }]
    }
  };
});

Multi-Turn Tool Calling Limitation

The batch API doesn't support multi-turn tool calling within a single request. This means you cannot:

If your workflow requires tool execution mid-processing, you must use the synchronous API. This limitation is a direct exam test point - if a scenario describes a batch workflow that needs to call external tools during processing, the correct answer is to use the synchronous API for that step.

Key Concept

The Message Batches API provides 50% cost savings with an up to 24-hour processing window and no latency SLA. Use it only for latency-tolerant workflows (overnight reports, weekly audits). Blocking workflows (pre-merge checks) must remain synchronous. Always refine prompts on a sample set before submitting large batches.

Prompt Optimisation Before Batch Submission

The most cost-effective batch processing strategy is to invest time in prompt refinement before submitting large volumes:

  1. Sample set testing: Take 5-10 representative documents covering the range of formats, edge cases, and document types in your batch
  2. Iterate on the sample: Refine your extraction prompts, add few-shot examples, adjust schema design until the sample set achieves high accuracy
  3. Submit the full batch: With refined prompts, your first-pass success rate will be significantly higher
  4. Handle failures: Resubmit only the failed documents with targeted modifications

This workflow slashes total cost. A 90% first-pass success rate on 1,000 documents means only 100 retries. A 60% first-pass rate means 400 retries, four times the resubmission cost, plus the batch processing cost for those retries.

Exam traps

Practice question

Your team wants to reduce API costs for automated analysis. You have two workflows: (1) a blocking pre-merge check that must complete before developers merge, and (2) a technical debt report generated overnight for review the next morning. Your manager proposes switching both to the Message Batches API for 50% cost savings. How should you evaluate this proposal?

  • A Switch both to batch processing with status polling to check for completion

    Status polling does not change the fundamental constraint: the batch API has no guaranteed latency SLA. Pre-merge checks cannot depend on a 24-hour processing window regardless of polling strategy.

  • B Use batch processing for the technical debt reports only; keep real-time calls for pre-merge checks Correct

    Pre-merge checks are blocking workflows - developers wait for results. The 24-hour batch processing window is unacceptable. Technical debt reports are overnight and latency-tolerant, making them ideal for batch processing at 50% savings.

  • C Keep real-time calls for both workflows to avoid batch result ordering issues

    Batch results are correlated using custom_id fields, so ordering is not an issue. The real concern is latency requirements, which this answer misidentifies.

  • D Switch both workflows to batch processing, with a timeout fallback to real-time if the batch takes too long

    This adds unnecessary complexity. The simpler and correct approach is to match each workflow to the appropriate API based on its latency requirements.

Build exercise: Design a Batch Processing Strategy

Intermediate · 45 minutes

You'll practice:

  1. List 5 workflows in a hypothetical organisation and categorise each as blocking (synchronous) or latency-tolerant (batch-eligible) with justification

    The matching rule between synchronous and batch API is the most tested concept in this task statement. The exam presents a scenario where a manager proposes switching everything to batch for cost savings, and you must identify which workflows cannot tolerate the 24-hour processing window.

    You should see: A table with 5 workflows, each clearly categorised with justification. Blocking workflows have someone or something waiting for the result. Batch-eligible workflows consume results later with no real-time dependency.

    Hints
    1. Think about who or what is waiting for the result. If a developer is blocked pending completion, it must be synchronous.
    2. Consider these workflow types: pre-merge CI checks, overnight reports, real-time chat responses, weekly audit summaries, nightly test generation. Classify each by latency tolerance.
    3. const workflows = [
        { name: "Pre-merge code review", type: "synchronous", reason: "Developer blocked pending merge approval" },
        { name: "Weekly technical debt report", type: "batch", reason: "Consumed Monday morning, no real-time dependency" },
        { name: "Real-time customer support", type: "synchronous", reason: "Customer waiting for response" },
        { name: "Nightly test generation", type: "batch", reason: "Results consumed next business day" },
        { name: "Overnight document extraction", type: "batch", reason: "Results processed in morning batch" }
      ];
  2. Define a batch submission for 20 documents using the Message Batches API format with unique custom_id fields for each document

    custom_id fields are the mechanism for correlating request-response pairs in batch results. Without unique identifiers, you cannot determine which documents succeeded or failed, making failure handling impossible.

    You should see: A valid batch request object with 20 entries, each containing a unique custom_id, model specification, max_tokens, and a messages array with the document content.

    Hints
    1. Use a naming convention for custom_id that encodes the document type and index for easy identification in results.
    2. Each request in the batch gets its own params object with model, max_tokens, and messages. The custom_id is the correlation key.
    3. const batchRequest = {
        requests: documents.map((doc, i) => ({
          custom_id: `doc-${doc.type}-${i.toString().padStart(3, "0")}`,
          params: {
            model: "claude-sonnet-5",
            max_tokens: 4096,
            messages: [{ role: "user", content: doc.content }]
          }
        }))
      };
      
      const batch = await client.batches.create(batchRequest);
  3. Implement failure handling: parse batch results, identify failures by custom_id, and construct a retry batch containing only failed documents with increased max_tokens

    Resubmitting only failures with targeted modifications is the correct batch failure pattern. Resubmitting the entire batch wastes cost on already-successful documents. The exam tests that you understand custom_id correlation and targeted retry.

    You should see: A failure handler that filters results by error status, extracts the custom_id values of failures, looks up the original documents, and creates a retry batch with modifications like increased max_tokens or chunked content.

    Hints
    1. Filter batch results by result.type to identify errored entries. Use the custom_id to look up the original document for resubmission.
    2. Common retry modifications include increasing max_tokens for truncated outputs, chunking oversized documents, and adding format-specific few-shot examples.
    3. const results = await client.batches.results(batch.id);
      const failures = results.filter(r => r.result.type === "errored");
      const failedIds = failures.map(f => f.custom_id);
      
      const retryBatch = {
        requests: failedIds.map(id => {
          const originalDoc = documentsById[id];
          return {
            custom_id: `${id}-retry-1`,
            params: {
              model: "claude-sonnet-5",
              max_tokens: 8192,
              messages: [{ role: "user", content: chunkIfNeeded(originalDoc) }]
            }
          };
        })
      };
  4. Calculate the batch submission frequency needed to guarantee a 30-hour SLA given the 24-hour maximum processing window

    SLA calculation with the 24-hour batch processing window is a direct exam test point. You must work backwards from the SLA deadline to determine when to submit, accounting for the maximum processing time plus a safety margin.

    You should see: A calculation showing: 30-hour SLA minus 24-hour maximum processing window equals 6 hours of buffer. Submission must occur at least 30 hours before the deadline, with batches submitted every 4-6 hours to guarantee the SLA with margin.

    Hints
    1. Work backwards from the deadline. If the report is due at 09:00 Monday, when is the latest you can submit the batch?
    2. Account for the worst case: the batch takes the full 24 hours. Your submission must happen early enough that even worst-case processing completes before the deadline.
    3. // SLA calculation
      const slaHours = 30;
      const maxProcessingHours = 24;
      const bufferHours = slaHours - maxProcessingHours; // 6 hours
      
      // If report due Monday 09:00:
      // Latest submission: Sunday 03:00 (30 hours before)
      // Recommended: Submit every 4-6 hours for safety
      // Submission schedule: Saturday 21:00, Sunday 03:00 (backup)
      
      const submissionFrequencyHours = Math.floor(bufferHours * 0.75); // 4.5 hours
      console.log(`Submit every ${submissionFrequencyHours} hours to guarantee SLA`);
  5. Create a 5-document sample set and refine extraction prompts iteratively before submitting the full batch of 20 documents

    Prompt refinement on a sample set before batch submission is the most cost-effective batch processing strategy. A 90% first-pass success rate means 2 retries on 20 documents. A 60% first-pass rate means 8 retries, four times the resubmission cost.

    You should see: A sample set covering the range of document types and edge cases, 2-3 prompt iterations improving accuracy on the sample, and then the full batch submission achieving a high first-pass success rate.

    Hints
    1. Select sample documents that cover the range of formats, edge cases, and document types in your full batch, not just the easy ones.
    2. Track first-pass success rate on each iteration. Stop refining when the sample achieves above 90% accuracy, then submit the full batch.
    3. // Select representative sample
      const sampleSet = selectRepresentativeSample(documents, 5);
      
      // Iterate on sample
      let prompt = initialPrompt;
      for (let iteration = 0; iteration < 3; iteration++) {
        const sampleResults = await runBatch(sampleSet, prompt);
        const accuracy = calculateAccuracy(sampleResults);
        console.log(`Iteration ${iteration}: ${accuracy}% accuracy`);
        if (accuracy >= 90) break;
        prompt = refinePrompt(prompt, sampleResults);
      }
      
      // Submit full batch with refined prompt
      const fullBatch = await client.batches.create(
        buildBatchRequest(documents, prompt)
      );

Sources