Every endpoint that survives contact with the public eventually needs a limiter. The interesting part is not whether to have one. It is which shape you choose, because the three common choices behave very differently on the only traffic that matters: the burst.
I run PostEngage on my own, where every outbound message goes through an API with its own limits, and every external call is a job with an explicit retry, dead-lettering and an alert. Being on both sides of a rate limiter — enforcing one, and staying inside somebody else's — makes you care about a detail most posts skip, which is what the limiter tells the client when it says no.
What a token bucket actually is
Three rules, and they fit on a napkin.
Stripe's engineering post Scaling your API with rate limiters describes the implementation in one sentence: "a centralized bucket host where you take tokens on each request, and slowly drip more tokens into the bucket. If the bucket is empty, reject the request."
The two numbers are independent, which is the whole appeal. Capacity 100 with a refill of 50 a second means a client that has been quiet can fire 100 requests immediately, then settles to 50 a second for as long as it keeps pushing. Idle time banks tokens, but only up to the brim, so a client cannot save up an hour of quiet and spend it in one second. A bucket that has been emptied is full again after capacity ÷ refill seconds, which here is two.
That "forgive the burst, hold the average" behaviour is what makes buckets pleasant to be a client of. Real traffic is bursty: a page load fires six API calls at once, a sync job wakes up, a user hammers refresh. A limiter that only enforces the average punishes normal behaviour.
One minute of traffic
Here is the default run from the simulator below: 60 seconds at 40 requests a second, with a spike to 200 between the 20 and 30 second marks. The bucket holds 100 tokens and refills at 50 a second.
The numbers: 4,000 requests were offered, 2,595 were served and 1,405 were refused. The busiest single second let through 145 requests, which is nearly three times the sustained limit of 50, and that is the bucket doing its job rather than failing at it.
Watch the token level in the lower panel. It sits at the brim during the quiet period, falls to zero almost immediately when the spike starts, and only recovers once demand drops below the refill rate. A bucket protects you from a short burst. It does not protect you from sustained overload, and nothing that answers requests can.
The two rivals
Give all three limiters the same long-run allowance — 50 a second, which for the window limiters is 500 per 10-second window — and feed them identical traffic.
| Property | Token bucket | Fixed window | Sliding window |
|---|---|---|---|
| Burst allowance | Up to capacity, then the refill rate | Whatever is left in the current window | None beyond the limit |
| Boundary problem | No | Yes: up to 2× the limit across a reset | No |
| State per client | A token count and a timestamp | A counter with an expiry | A timestamp per request, or per sub-window |
| In this run, allowed | 2,595 | 2,500 | 2,500 |
| In this run, busiest second | 145 | 200 | 140 |
Fixed windows are the cheapest thing that works, which is why they are everywhere. Redis documents the canonical implementation in the INCR command reference: a key per client per second, incremented on each call with an expiry attached, rejecting when the counter passes the limit. The docs are also honest about the sharp edge in the simpler variant, where a client that increments but fails to set the expiry leaks the key: "In the above code there is a race condition", fixed by moving the increment and expiry into a Lua script so they run together.
Sliding windows remove the boundary effect by counting over a trailing period rather than a clock-aligned one. The exact version keeps a timestamp per request, which is accurate and expensive for a busy client; the common compromise weights the previous window's count by how far into the current one you are.
The boundary problem, concretely
This is the failure mode that makes fixed windows unsuitable for protecting anything fragile.
A limit of 100 a minute, with 100 requests at 00:59 and 100 more at 01:00, is two compliant windows and one very uncompliant two seconds. Nothing in the counter notices. If your limit exists to protect a database connection pool or a downstream API, the number that matters is the instantaneous rate, and the fixed window does not bound it.
quick check
A fixed window allows 100 requests a minute. A client sends 100 requests at 00:59 and another 100 at 01:00. What happens?
Each window counted 100 requests and allowed them. The policy was respected on paper while the server received twice its intended rate in a two-second span. A token bucket bounds this to its capacity; a sliding window bounds it to the limit.
Run it yourself
Change the capacity, the refill rate and the shape of the burst, and switch between the three limiters on the same traffic.
simulator · 60 seconds of traffic
What a rate limiter does to a burst
requests a second, allowed and rejected
| limiter | allowed | rejected | busiest second |
|---|
Arrivals are evenly spread within each second, so this is kinder than real traffic, which clumps. All three limiters are given the same long-run allowance: the windows use refill × window as their limit. The simulation is one limiter for one client; a shared limiter across several servers needs a central counter, which adds a round trip and a failure mode of its own.
The setting worth playing with is capacity. Raise it and the busiest second climbs while the total served barely moves: you are deciding how much of a burst reaches whatever is behind the limiter.
Saying no properly
A rate limiter is a contract, and half of it is what you return. The status code has been standardised since 2012 in RFC 6585: "The 429 status code indicates that the user has sent too many requests in a given amount of time ('rate limiting')." The same section adds two things people forget: "The response representations SHOULD include details explaining the condition, and MAY include a Retry-After header indicating how long to wait before making a new request," and "Responses with the 429 status code MUST NOT be stored by a cache."
Telling the client when to come back matters more than the code. Without it, a well-behaved client has to guess, and a badly behaved one retries immediately and makes your overload worse.
The newer work is the IETF's RateLimit header fields draft, which standardises the quota headers that every API had been inventing separately. It defines RateLimit-Policy for advertising the policy and RateLimit for the live state, as structured fields:
RateLimit-Policy: "burst";q=100;w=60,"daily";q=1000;w=86400
RateLimit: "default";r=50;t=30The first says there are two policies: 100 requests per 60 seconds, and 1,000 per day. The second says 50 remain in the current policy and it resets in 30 seconds. The draft is deliberately neutral about how you count — "This specification does not mandate a specific throttling algorithm" — so a token bucket can advertise its state as easily as a window can. It also ties the two mechanisms together: if you send both, "the Retry-After field value SHOULD NOT reference a point in time earlier than the end of the effective window".
Send the headers on successful responses too, not only on the 429. A client that can see it has ten requests left will slow down; a client that only finds out by being rejected cannot.
Where to put it, and what to key it on
Stripe's post is useful here because it separates limiters by what they protect rather than by algorithm. Alongside the per-user request rate limiter it describes a concurrent requests limiter — "You can only have 20 API requests in progress at the same time" — and two load shedders, one that reserves capacity for critical traffic ("We always reserve a fraction of our infrastructure for critical requests") and one that sheds lower-priority work when workers back up.
That layering is the practical lesson. A single per-user limit does not protect you from one expensive endpoint, and no per-user limit protects you from a thousand users arriving at once.
- Key on the thing you are protecting. For a paid API that is the account or tenant, not the IP. For sign-in and password reset it is the IP and the account, because the attacker controls one of them.
- Limit concurrency as well as rate. Ten slow requests at once can hurt more than a hundred fast ones, particularly when each holds a database connection.
- Put cheap limits at the edge and precise ones in the app. A per-IP limit at the proxy costs nothing; the per-tenant quota needs application context.
- Expect the shared counter to be a dependency. A distributed limiter is a Redis round trip on every request, and you must decide in advance whether it fails open or closed. Fail open and an outage removes your protection; fail closed and a Redis blip becomes a total outage.
- Give background work its own budget. A bulk export should not be able to spend the interactive quota, which is the same reasoning as keeping a backlog from eating the system.
- Make retries safe before you make them polite. Clients will retry your 429s, so the endpoints behind them need idempotency.
If you are on the receiving end of somebody else's limiter — as anyone sending on the WhatsApp Business API is — the same maths runs in reverse. Read their published limits, keep your own token bucket slightly below them, and honour Retry-After rather than discovering their enforcement the hard way.
questions people ask
What is the token bucket algorithm?
A counter of permits that refills at a fixed rate up to a maximum. Each request spends one token, and requests arriving at an empty bucket are rejected. Capacity sets the largest burst allowed; the refill rate sets the sustained limit.
Token bucket or leaky bucket?
A token bucket allows bursts up to its capacity and then enforces the rate. A leaky bucket drains at a constant rate and so smooths output completely, which is what you want for shaping traffic you are sending, rather than policing traffic you are receiving.
What is wrong with a fixed window rate limiter?
The counter resets on a clock boundary, so a client can use a full window at the end of one period and another full window at the start of the next, hitting up to twice the intended rate in a short span.
What status code should a rate limiter return?
429 Too Many Requests, per RFC 6585, with a body explaining the condition and a Retry-After header. Responses with 429 must not be cached.
What are the RateLimit headers?
Standardised fields from the IETF HTTPAPI working group: RateLimit-Policy advertises the quota policy and RateLimit reports the remaining quota and reset time. They are algorithm-neutral, so they work for buckets and windows alike.
Should a rate limiter queue requests instead of rejecting them?
Usually not. Queueing converts excess load into latency and memory, and upstream timeouts then produce retries that add more load. Reject early, tell the client when to return, and keep any deliberate smoothing tightly bounded.
The short version
A token bucket is two numbers: capacity, the burst you forgive, and refill rate, the limit you enforce. It suits real traffic, which arrives in clumps, and it bounds the instantaneous rate in a way a fixed window does not. Fixed windows are cheap and let a client send double the limit across a reset; sliding windows close that gap and cost more state.
Pick the bucket unless you have a reason not to, size the capacity as the burst you are willing to pass downstream, and put as much care into the refusal as the enforcement: 429, a Retry-After that means something, and RateLimit headers on every response so clients can pace themselves instead of guessing.