5.3 - Stakeholder Feedback Loops and Expectation and SLA Alignment

Build exercise: Write an expectations and SLO charter with a feedback loop

Advanced · 60 minutes

You'll practice:

• Define service, quality and cost SLIs and SLOs for a probabilistic system
• Derive a service objective from documented dependency behaviour and resilience design
• Design feedback channels that feed a versioned eval set, with owners and cadence
• Specify how disagreement and scope change are handled
• Compute a consistency metric (pass^k) from repeated trials

1. For the claim-email summariser (or a scenario of your choice), draft an SLI/SLO table with at least two service, three quality and one cost indicator. For each, state the measurement method, the data source, the target as a placeholder to be set from evidence, and who signs it off.

   Separating layers prevents a single vanity metric and forces you to say how each number would actually be measured.

   You should see: A table of at least six SLIs, none of which is 'accuracy' without a definition, each with a method and an owner.

   Hints

   1. If this number went bad, which user-visible problem would it explain?

   2. Take service SLIs from request success and latency, quality SLIs from labelled samples, reviewer overrides and schema validity, and cost from spend per completed task.

   3. Example row: 'Quality: summaries matching the senior handler's label on the weekly 100-email sample; target set after baseline run; owner: head of claims; measured by blind double review.'

2. Write a one-page 'dependency reality' section: list the API errors you must handle (429 with and without retry-after, 500, 529, timeouts), what your system does for each, and your degraded mode. Read the current Anthropic errors, rate-limit and service-tier pages first and note anything your SLO must not assume.

   This turns 'we depend on a vendor' into concrete engineering and shows what you can and cannot promise externally.

   You should see: A table mapping each failure class to retry policy, fallback and user impact, plus a list of assumptions you deliberately did not make (for example, guaranteed capacity).

   Hints

   1. Which failures clear on their own, and which need a human or a budget change?

   2. Retry with backoff on transient errors, honour retry-after when present, treat a spend-cap 429 as an alert rather than a retry case, and define what the user sees when the model path is down.

   3. Example: 'Spend-cap 429 (no retry-after): stop retrying, page the owner, switch to manual queue, notify support; expected duration: until cap raised or next billing period.'

3. Define the error budget policy. State what happens when the quality SLO or the service SLO budget is exhausted (for example, freeze prompt and model changes, prioritise fixes, increase review sampling), who has authority to invoke it, and how product, engineering and operations agree the numbers.

   An SLO without a consequence for missing it is only a wish. The policy is what aligns stakeholder incentives.

   You should see: A half-page policy with triggers, actions, an authority and a review cadence.

   Hints

   1. What would you actually do differently the week after the budget is gone?

   2. Tie actions to each layer: quality breach means tighter review and no new features; service breach means reliability work; cost breach means model or effort review.

   3. Example: 'If the override rate exceeds the threshold for two consecutive weeks, freeze prompt changes except fixes, raise human review sampling to 100 percent for the affected category, and review at the Friday quality meeting.'

4. Design the feedback loop: list four channels (for example reviewer overrides, end-user ratings, support tickets, weekly sample grading), and for each give the owner, cadence, and where the data goes. Then write the rule that every prompt or configuration change must pass the regression eval before release. Include the procedure for a scope-change request.

   This makes feedback an engineering input rather than anecdote and closes the loop with stakeholders.

   You should see: A channel table, a change gate, and a scope-change procedure that starts with a lightweight discovery and ends with an updated eval set and decision record.

   Hints

   1. Where does a single bad output end up in your system, and who sees it?

   2. Every confirmed failure becomes a test case; every change runs the regression set; every scope change triggers a mini-discovery, eval extension and stakeholder sign-off.

   3. Example: 'Support tags a ticket ai-error -> triager reproduces within two working days -> case added to eval set with expected label -> fix proposed -> regression run -> release note to stakeholders.'

5. Implement a tiny consistency calculation. Given repeated trials per task (each a list of booleans), compute the fraction of tasks where all k trials passed (pass^k) and the fraction where at least one passed (pass@k), and compare them for k=1, 3, 5 using invented data. Explain in two sentences which figure belongs in a customer-facing reliability statement.

   It shows why average or best-case metrics can overstate reliability, which is a common source of over-promising.

   You should see: For invented data with moderate per-trial success, pass@k rising toward 1 as k grows while pass^k falls, and a clear explanation of why the latter reflects consistency.

   Hints

   1. What does 'at least one attempt succeeded' hide from a customer who only sees one attempt?

   2. Group trials by task, then apply all() for pass^k and any() for pass@k over the first k trials.

   3. def pass_stats(results, k):
          tasks = list(results.values())
          all_k = sum(all(t[:k]) for t in tasks) / len(tasks)
          any_k = sum(any(t[:k]) for t in tasks) / len(tasks)
          return all_k, any_k
      
      results = {'t1': [True, True, False, True, True], 't2': [False, False, True, True, False], 't3': [True] * 5}
      for k in (1, 3, 5):
          print(k, pass_stats(results, k))