1.11 - Versioning and Rolling Out Model Changes

Build exercise: Pin a model version, build a format-focused eval set, and simulate a staged rollout comparison

Intermediate · 35 minutes

You'll practice:

• Explicitly pin a model version in configuration
• Build an eval set that specifically checks format adherence, not just general quality
• Run the same eval set against two versions to compare before a rollout decision

1. In your config, replace any "latest"-style model reference with an explicit pinned version string, stored alongside your other config values from Lesson 1.9.

   This is the change that gives you control over when behaviour shifts, instead of inheriting it silently.

   You should see: The application behaves identically across repeated runs and deploys, since the version is now fixed.

   Hints

   1. Where should this pinned version live relative to the rest of your API key and generation-parameter config?

   2. Keep the pinned version string in the same config source as your API key and max_tokens/temperature values, not hardcoded separately somewhere else in the codebase.

   3. // config.js
      export const MODEL = process.env.CLAUDE_MODEL_VERSION ?? "claude-sonnet-5-20260115";

2. Write 8-10 eval cases specifically targeting the exact output format your feature depends on (e.g. a strict JSON schema, a required field set, a particular structure), not just general correctness.

   This is the specific gap the lesson calls out - a general quality eval set can miss a format regression entirely.

   You should see: A list of prompts each paired with a schema-conformance check (does the response parse as valid JSON, are all required fields present) rather than a subjective quality judgment.

   Hints

   1. If your feature's real failure mode is a downstream parser breaking, what should each eval case check for, beyond "is the answer good"?

   2. Each eval case should assert something mechanically checkable - valid JSON, required keys present, correct types - so a regression is caught by a pass/fail check, not a subjective read.

   3. const evalCases = [
        { prompt: "Extract name and age from: 'Sam is 34'", check: (out) => { const j = JSON.parse(out); return typeof j.name === "string" && typeof j.age === "number"; } },
        // ... 7-9 more covering edge cases like missing data, unusual phrasing, multiple entities
      ];

3. Run the full eval set against your currently pinned version and against a newer candidate version, and compare pass rates before deciding whether to roll forward.

   This is the actual pre-rollout gate the lesson describes - comparing two versions on the same format-focused eval set rather than trusting a general impression that the new version is 'better'.

   You should see: A pass-rate comparison between the two versions on your format-focused eval set, surfacing any case where the newer version regresses even if its answers are otherwise more fluent.

   Hints

   1. If the newer version writes better prose but breaks the JSON schema on two of your ten cases, should you roll it out?

   2. Report pass/fail per case for both versions side by side - a version with better general writing but a lower schema-conformance pass rate is a regression for this feature, not an upgrade.

   3. for (const version of [currentVersion, candidateVersion]) {
        let passed = 0;
        for (const c of evalCases) {
          const out = await callModel(version, c.prompt);
          if (safeCheck(c.check, out)) passed++;
        }
        console.log(version, "pass rate:", passed, "/", evalCases.length);
      }