Applications & Integration · Lesson 8 of 11

1.8 - Rate Limits, Retries and Idempotency

Handle 429s and transient failures the way a production system should: backing off, not hammering the API harder.

API usage is limited on more than one axis at once — requests per minute, and input and output tokens per minute tracked separately — and whichever limit you hit first is the one that returns a 429. Limits scale with account usage tier, so a low-volume developer account and a high-volume production account can have very different ceilings for the same model. A well-behaved client reads the rate-limit headers the API returns on every response (current limit, remaining quota, and a reset time) to anticipate throttling before it happens, and backs off exponentially with jitter when a 429 does occur, rather than retrying immediately in a tight loop, which only makes the throttling worse for that client and for others sharing infrastructure.

Transient vs. permanent failures

Not every non-200 response should be retried. A 429 (rate limited) or a 5xx/overloaded_error (server-side, transient) is worth retrying with backoff. A 400 (malformed request), 401 (bad credentials), or 403 (unauthorised) will fail identically every time on an unchanged payload and should surface as a bug for a human to fix, not be retried into a loop that burns time and quota for no benefit. Most official SDKs implement this distinction automatically — a default retry policy (commonly two automatic retries with backoff) that applies only to the genuinely retryable status codes, which is worth knowing both so you don't duplicate the logic and so you know what to override when your application needs a different policy.

Idempotency

A retried request risks being processed twice if the original request actually succeeded server-side but the client never received the response (a dropped connection, a client-side timeout). For calls where a duplicate execution matters — anything that triggers a side effect through a tool, or a request tied to billing on your own downstream system — supplying an idempotency key lets a retried request with the same key be recognised as a duplicate rather than executed again. This is a general API-reliability pattern, not unique to Claude, but it's exactly the kind of nuance the exam expects you to connect to retry logic rather than treat as a separate topic.

Common exam distractor

Retrying a 400 error is a classic wrong answer. A malformed request is a client-side bug — retrying the identical broken payload produces the identical error, forever, and burns retry budget that should have gone toward a request that could actually succeed.

Backoff mechanics

Exponential backoff means each retry waits roughly double the previous delay (e.g. 1s, 2s, 4s, 8s), capped at some maximum so a chain of failures doesn't eventually wait minutes between attempts. Jitter — adding a small random offset to each computed delay — exists specifically to prevent many clients that all got rate-limited at the same moment from retrying in lockstep and re-triggering the same throttle together; without jitter, a fleet of workers can synchronise into a thundering herd that keeps re-hitting the limit at the same instant, forever.

Limits scale with usage tier, and service tiers change the trade-off

An account's requests-per-minute and tokens-per-minute ceilings increase as usage and account standing grow — a newly created account has materially lower limits than an established production account, which matters when interpreting a 429 during early testing versus one that appears after a traffic spike on an established integration. Separately, some workloads can choose between a standard service tier and a priority tier that trades a cost premium for a stronger throughput guarantee and reduced likelihood of being throttled during periods of high overall demand — worth knowing as a lever distinct from backoff logic itself: backoff handles a rate limit gracefully once it happens, while service tier selection is about reducing how often it happens in the first place for latency-sensitive workloads.

Concurrent connections as a separate limit

Beyond requests-per-minute and tokens-per-minute, some accounts are also bounded on the number of concurrent open connections or in-flight requests at any one instant — a distinct axis from the rate-over-time limits, and one that shows up specifically in a client that fires off a large batch of requests in parallel rather than at a steady rate. A worker pool that opens far more simultaneous connections than the account supports can get throttled even while comfortably under the per-minute request and token ceilings, which is a useful thing to check before assuming a 429 must be a rate-over-time problem.

Key concept

A large parallel batch job should throttle its own concurrency deliberately (a worker pool with a fixed max, not "fire everything at once") rather than relying entirely on retry logic to absorb the resulting 429s after the fact. Preventing avoidable throttling is cheaper than recovering from it.

Exam traps

Applying the same exponential-backoff retry logic to every non-200 status code, including 400s.

A 400 means the request itself is invalid; retrying it unchanged just repeats the same failure and wastes time that should go to fixing the request.
Retrying failed requests immediately in a tight loop with no delay.

Immediate retries under an active rate limit typically extend the throttling rather than resolving it - the client needs to back off, not hammer the API harder.
Implementing backoff without jitter, using a fixed delay sequence identical across all retrying clients.

Without jitter, many clients rate-limited at the same moment retry in lockstep, re-triggering the same throttle together - a thundering herd. A small random offset per client breaks that synchronisation.
Assuming a retried request can never be processed twice, so no idempotency handling is needed for retries on requests with side effects.

A request can succeed server-side while the client never receives the response, due to a dropped connection or timeout - a naive retry in that case duplicates the side effect. An idempotency key lets a retry be recognised as a duplicate instead of re-executed.

Practice question

A batch job hits 429 responses partway through a large run. What is the correct handling?

A Immediately retry each failed request in a tight loop until it succeeds.
Retrying immediately under an active rate limit typically extends the throttling rather than resolving it.
B Abandon the batch job entirely and require a human to restart it from scratch.
A 429 is a transient, expected condition on high-volume work - it doesn't warrant abandoning the job.
C Back off exponentially with jitter before retrying the throttled requests, resuming normal pace once they succeed. Correct
This is the standard correct handling for a rate-limit response: give the limit time to reset, then continue.
D Switch to a smaller model, since rate limits only apply to larger models.
Rate limits apply per account/tier across usage; switching models doesn't inherently avoid them.

Build exercise: Implement exponential backoff with jitter and an idempotency key

Intermediate · 35 minutes

You'll practice:

Distinguish retryable from non-retryable errors
Implement exponential backoff with jitter
Read rate-limit headers proactively
Apply an idempotency key to a retried side-effecting call

Write a small wrapper function around your API call that retries on 429 and 5xx responses only, with delay doubling each attempt plus a small random jitter, up to a fixed max attempt count.

This is the actual pattern a production client needs, not just a concept to recognise on the exam.

You should see: A 400 or 401 fails immediately with no retry; a simulated 429 retries with increasing, jittered delay.

Hints

Which status codes belong in your retryable set, and which don't, based on whether the same request would succeed unchanged?

Cap the number of retries - unbounded retry is its own failure mode. Jitter (a small random offset added to each delay) prevents many clients from retrying in lockstep and re-triggering the same throttle together.

async function withRetry(fn, maxAttempts = 5) {
  for (let attempt = 0; attempt < maxAttempts; attempt++) {
    try { return await fn(); }
    catch (err) {
      if (![429, 500, 502, 503, 529].includes(err.status)) throw err;
      const delay = Math.min(1000 * 2 ** attempt, 30000) + Math.random() * 500;
      await new Promise(r => setTimeout(r, delay));
    }
  }
  throw new Error("max retries exceeded");
}

Log the rate-limit headers (limit, remaining, reset) from a real response, and add a check that proactively slows down before you hit zero remaining, rather than waiting for a 429.

Reacting only after a 429 already happened is one step behind what a well-behaved client does - the headers tell you it's coming.

You should see: A log line showing decreasing remaining-quota values across several calls, and a deliberate pause inserted once remaining drops below a small threshold.
Hints
1. ```
Where do rate-limit values live on the response - the body, or somewhere else?
```
2. ```
Rate-limit info arrives as response headers, not in the JSON body - read them from the raw HTTP response object your client library exposes.
```
3. ```
console.log(response.headers['anthropic-ratelimit-requests-remaining']);
if (Number(response.headers['anthropic-ratelimit-requests-remaining']) < 5) {
  await sleep(2000); // proactive slowdown before hitting zero
}
```
Simulate a client timeout on a request that actually succeeded server-side (e.g. by cutting the connection after sending but before reading the response in a test harness), then retry it with an idempotency key and confirm it isn't double-processed.

This exercises the specific failure mode idempotency keys exist for - a request that succeeded but whose response the client never saw.

You should see: The retried request with the same idempotency key is recognised as a duplicate of the original rather than triggering a second execution.
Hints
1. ```
What single header, sent identically on the original and the retry, is what lets the server recognise a duplicate?
```
2. ```
Generate one idempotency key per logical operation before the first attempt, and reuse that exact same key on every retry of that same operation - a new key per retry defeats the purpose.
```
3. ```
const idempotencyKey = crypto.randomUUID();
await withRetry(() => client.messages.create(params, { idempotencyKey }));
// Same idempotencyKey reused across all retries of this one logical call
```

Sources

Anthropic docs - Errors and rate limits
Anthropic docs - SDK retry behaviour