3.3 - A/B Testing and Iterative Improvement

Build exercise: Plan and simulate a safe A/B comparison: sticky bucketing, a paired offline comparison and a rollout rule

Advanced · 75 minutes

You'll practice:

• Write an experiment plan with hypothesis, randomisation unit, primary metric, guardrails, stopping rule and rollback trigger
• Implement deterministic, sticky bucketing that supports a small canary share
• Run a paired offline comparison and quantify uncertainty with a bootstrap interval instead of a single delta
• Define a ship or rollback decision rule that includes guardrail metrics and cache warm-up

1. Choose a change you might really make (a shorter prompt, a cheaper model tier, a lower effort setting). Write an experiment plan covering: the single variable being changed, the hypothesis, the randomisation unit, the primary metric, at least three guardrail metrics with tolerances, how you will decide the duration, and the rollback trigger.

   Writing the decision rule before running anything is what prevents post-hoc rationalisation. It is also the artefact an exam scenario expects you to be able to produce.

   You should see: A one-page plan in which every guardrail has a stated tolerance, the change is a single variable, and the duration or sample is justified by the baseline rate and the smallest effect you care about (not a rule of thumb).

   Hints

   1. If this change went badly wrong, which user-visible metric would move first, and which would move last?

   2. List the primary metric and the ways the change could hurt others: latency, cost per completed task, safety flags, escalation to humans, error rate. Give each a tolerance you would accept.

   3. Example: Variable: model tier only (prompt unchanged). Hypothesis: candidate keeps resolution rate within the agreed margin while lowering cost per resolved conversation. Unit: user, sticky. Primary: resolution rate. Guardrails: safety-flag rate (no increase beyond tolerance), escalation rate, p95 latency, error rate, cost per resolved conversation. Duration: from a power calculation using the current resolution rate and the smallest drop we would accept; no early stop. Rollback: any guardrail beyond tolerance for two consecutive windows, or any critical safety incident.

2. Implement deterministic bucketing: given an experiment name and a user id, return control or candidate with a configurable canary share, so the same user always lands in the same arm and changing the experiment name reshuffles users. Test that the observed split is close to the configured share over many ids.

   Sticky assignment prevents contamination in multi-turn products, and hashing on experiment name plus user id makes the assignment reproducible without storing state.

   You should see: A function that returns the same arm for the same inputs on every call, and a quick simulation over thousands of ids whose split is near the configured share.

   Hints

   1. Why would calling random.random() per request be a problem for a conversation that spans several requests?

   2. Hash the string experiment:user_id to a number in [0, 1) and compare it to the cumulative arm weights.

   3. import hashlib
      
      def bucket(experiment: str, user_id: str, arms=("control", "candidate"), weights=(0.95, 0.05)) -> str:
          h = int(hashlib.sha256(f"{experiment}:{user_id}".encode()).hexdigest(), 16) % 10_000 / 10_000
          cum = 0.0
          for arm, w in zip(arms, weights):
              cum += w
              if h < cum:
                  return arm
          return arms[-1]
      
      counts = {}
      for i in range(20_000):
          a = bucket("tier-swap-01", f"user-{i}")
          counts[a] = counts.get(a, 0) + 1
      print(counts)   # candidate share should be close to 5%

3. Run the same evaluation set through the current and candidate configurations (same dataset, grader and judge; only the one variable differs), three trials each. Compute the per-case score difference and a bootstrap 95% interval for the mean difference. Report it per slice as well as overall.

   A paired comparison with an interval shows whether an apparent gain is distinguishable from noise, and per-slice results reveal regressions the overall mean hides.

   You should see: For the overall set and each slice: a mean difference, a lower and upper bound, and a plain-language reading (clear gain, clear loss, or not distinguishable).

   Hints

   1. If the interval for the mean difference includes zero, what is the honest conclusion?

   2. Average each configuration's trials per case first, subtract per case, then resample those per-case differences with replacement many times and take percentiles of the resampled means.

   3. import random
      
      def paired_bootstrap(a_scores, b_scores, n=5000, seed=1):
          """a_scores, b_scores: per-case mean scores in the same case order."""
          rng = random.Random(seed)
          diffs = [b - a for a, b in zip(a_scores, b_scores)]
          means = sorted(
              sum(rng.choice(diffs) for _ in diffs) / len(diffs) for _ in range(n)
          )
          return sum(diffs) / len(diffs), means[int(0.025 * n)], means[int(0.975 * n) - 1]
      
      delta, lo, hi = paired_bootstrap(control_by_case, candidate_by_case)
      print(f"mean diff {delta:+.3f}  95% interval [{lo:+.3f}, {hi:+.3f}]")

4. Compare cost and latency for the two arms from recorded usage. Separate the first requests (cold cache) from later requests (warm cache) and report cache_read_input_tokens as a share of total input tokens for each arm. State whether any cost difference is explained by cache warmth rather than by the variable under test.

   Prompt variants warm their caches independently, so an unfair comparison can make a good candidate look expensive or slow. Controlling for it is the Claude-specific part of A/B analysis.

   You should see: A table of cost per completed task and p95 latency for warm-cache requests only, plus each arm's cache-read share, and one sentence on how much of any gap disappears after warm-up.

   Hints

   1. If the candidate arm only receives a small share of traffic, how often will its cached prefix expire between requests?

   2. Compute total input as input_tokens + cache_creation_input_tokens + cache_read_input_tokens for each request, then the cache-read share per arm, and repeat the cost comparison on requests after each arm's first cache write.

   3. def cache_read_share(records):
          total = sum(r["input_tokens"] + r["cache_write"] + r["cache_read"] for r in records)
          return sum(r["cache_read"] for r in records) / total if total else 0.0
      
      warm = [r for r in arm_records if not r["is_first_in_window"]]   # exclude cold-start requests
      print("cache-read share, warm requests:", round(cache_read_share(warm), 3))

5. Write the final ship, hold or roll back decision rule as a table: for each metric, the observed result, the tolerance, and the verdict. Then write the rollout schedule (shadow, canary, ramp steps) and the exact condition that triggers automatic rollback.

   The decision table forces an explicit trade-off between the primary metric and guardrails, and a pre-agreed rollback removes argument during an incident.

   You should see: A completed table with a verdict per metric and an overall decision, and a rollout schedule whose every step names the gate that must pass before the next.

   Hints

   1. If the primary metric improved but one guardrail moved outside tolerance, is the overall answer ship, hold or roll back, and who owns that decision?

   2. A rule that reads 'ship only if the primary metric meets its margin and no guardrail is outside tolerance; otherwise hold and investigate' avoids ad hoc judgement.

   3. Example rollout: Step 1 shadow for the agreed window, gate: no format or error regressions. Step 2 canary share, gate: all guardrails within tolerance after cache warm-up. Step 3 stepwise ramp, gate: same, checked at each step. Rollback trigger: config flip to the previous prompt and model version if any guardrail is outside tolerance for two consecutive measurement windows.