Rate limit headers and backing off politely

What actually happens when you hit a rate limit

You send a request, and instead of a response body you get a 429. Nothing broke on your end. The model didn’t refuse the prompt. The provider’s gateway just decided you’ve sent too many requests, too many tokens, or opened too many concurrent connections in whatever window it’s tracking, and it’s telling you to slow down.

The instinct is to retry immediately. That’s the wrong move, and it’s worth understanding why before you write a single line of retry code.

A 429 is the provider’s load shedding mechanism doing exactly what it’s supposed to do. Every major LLM API, OpenAI, Anthropic, Google, enforces limits per organization or per API key, usually on at least two axes at once: requests per minute and tokens per minute. Some also cap concurrent requests separately from throughput. If you’re hammering the endpoint the moment you get throttled, you’re not recovering faster, you’re adding load to a system that just told you it’s already at capacity.

The headers worth reading

Most providers send you the state of your rate limit bucket on every response, not just the 429s. That’s the part people skip, because you only look at headers when something’s already on fire.

Anthropic’s API returns headers like anthropic-ratelimit-requests-limit, anthropic-ratelimit-requests-remaining, and anthropic-ratelimit-requests-reset, plus the same trio for tokens. OpenAI’s equivalent is x-ratelimit-limit-requests, x-ratelimit-remaining-requests, x-ratelimit-reset-requests, again mirrored for tokens. These show up on every successful call, not just the failures.

That means you can see a throttle coming before it happens. If remaining-requests drops to single digits and reset is 40 seconds out, you already know to slow down your next few calls instead of waiting for the 429 to tell you. This is the difference between a client that reacts and one that anticipates. If you’re running a pipeline that fires off batches, checking these headers after each call and throttling proactively is cheaper than eating rejected requests and re-sending them.

The other header that matters is Retry-After. When a 429 comes back, a well-behaved provider tells you exactly how many seconds to wait, sometimes as a header, sometimes in the error body. That number isn’t a suggestion. It reflects the actual state of the bucket you’re bound to. Ignoring it and retrying on your own schedule is how you end up looking like abuse traffic instead of a client that briefly went over budget.

Naive retry loops make things worse

The retry pattern I still see in production code, including code I’ve shipped and regretted, is: catch the exception, sleep a fixed amount, try again, repeat up to N times. Fixed sleep, say 2 seconds, is the problem. If your rate limit resets on a 60 second window and you’re retrying every 2 seconds, you’re sending roughly 30 requests during a period when the answer was always going to be no until the window rolls over. Every one of those retries counts against your request quota too, on some APIs, digging the hole deeper.

It gets worse under real load. If you’ve got ten worker processes all hitting the same rate limit and they all catch the 429 at roughly the same time, a fixed retry delay means they all retry at roughly the same time too. You get a synchronized wave of requests slamming into the endpoint right as it’s trying to recover, which is the thundering herd problem, and it can push you further into the 429 loop rather than out of it.

Exponential backoff with jitter, in practice

The fix is standard and well understood: exponential backoff with jitter. The shape is simple. First retry waits roughly 1 second, second retry waits roughly 2, third waits roughly 4, and so on, capped at some ceiling like 30 or 60 seconds so you’re not waiting five minutes on a call that’s timing out anyway. The jitter part means you don’t use that exact number, you randomize within a range around it, so ten workers hitting the same limit don’t all wake up and retry in the same instant.

A version that works well in practice: base delay times 2 to the power of the attempt number, then multiply by a random factor between 0.5 and 1.5. Cap attempts at 5 or 6 before you give up and surface the error, because past that point you’re usually dealing with something other than a transient rate limit, like an invalid key or a genuinely oversized request.

If you’re already reading the rate limit headers, you don’t need to guess the backoff at all on the first retry. If reset says 12 seconds, wait 12 seconds plus a small jitter, not some generic exponential guess. Fall back to exponential backoff only when you don’t have that information, like on a network timeout or a 500 where no rate limit header is present.

Respect Retry-After when it’s there

When a 429 includes a Retry-After value, use it as your floor, not a starting point to negotiate down from. I’ve seen code that takes Retry-After, decides it’s too conservative, and retries sooner anyway. That’s usually a mistake. The provider’s gateway knows the state of the bucket better than your client does. If it says 8 seconds, waiting 8 seconds and succeeding beats waiting 3 seconds and getting throttled again, because a second 429 often resets your position at the back of whatever queue or penalty window is in play.

If Retry-After is missing, that’s your signal to fall back to exponential backoff with jitter as described above, starting from a base of 1 to 2 seconds.

Concurrency limits are a different beast than request limits

It’s worth separating two failure modes that look identical from the outside but need different fixes. A requests-per-minute or tokens-per-minute limit is about volume over time, and backoff solves it. A concurrency limit, sometimes 400 or 5, sometimes higher depending on your tier, is about how many requests can be in flight at once, and backoff alone doesn’t fix that if your actual bug is that you’re firing off 200 async calls in a tight loop with no semaphore.

If you’re seeing 429s specifically tied to concurrency rather than volume, the fix is a semaphore or worker pool that caps in-flight requests to something comfortably under your limit, not a longer sleep. I’ve debugged this exact mistake in a batch summarization job: the retry logic was solid, exponential backoff, jitter, all of it, but the job still crawled because it kept re-queueing 40 concurrent calls against a limit of 20. The backoff was masking a concurrency bug instead of fixing it.

What this costs you if you get it wrong

Ignoring rate limits properly has a direct cost, not just a theoretical one. Failed requests that you retry badly still often count against your quota, so a bad retry loop can burn through your per-minute budget on rejected calls, leaving less room for the requests that would have actually succeeded. On a pipeline processing a queue of documents, a naive retry storm can turn a job that should finish in 10 minutes into one that takes an hour, because half the requests it’s making are 429s bouncing off a bucket that never gets a chance to refill.

There’s also a reputational cost with the provider, though it’s an unglamorous one. Sustained abusive retry patterns are exactly the traffic shape that trips secondary abuse detection on top of the ordinary rate limiter, and that can mean a more aggressive throttle applied to your key or org, not just the specific model call that failed.

A backoff worth shipping

The version of this I actually run in production: read the rate limit headers on every response and track remaining requests and tokens per key. On a 429, check for Retry-After first and honor it. If it’s absent, use exponential backoff starting at 1 second, doubling each attempt, jittered by plus or minus 50%, capped at 60 seconds, giving up after 5 attempts. Separately, cap concurrency with a semaphore sized comfortably under whatever the provider’s stated concurrent request limit is for your tier. None of this is exotic. It’s maybe 40 lines of code, and it’s the difference between a pipeline that degrades gracefully under load and one that falls over and pages someone at 2am.

If you’re building anything that calls an LLM API more than a handful of times a day, this is worth getting right once and reusing everywhere, rather than rewriting a slightly different broken version of it in every project.

More explainers like this live on the AI Tool Gazette home page.