Circuit breakers get configured from the library's example. Twenty requests, fifty per cent, five seconds — whatever was in the README. Then either it never opens and nobody notices, or it opens constantly and somebody sets the threshold higher until it stops.
Both outcomes come from the same gap: nobody works out how often ordinary noise will fill the window. It is a binomial tail probability, it takes a minute, and the answer is frequently alarming.
The arithmetic nobody runs
A breaker watches a rolling window of n requests and opens when k of them have failed. A healthy dependency still fails sometimes — a timeout, a retryable error, a pod being replaced — at some baseline rate p.
The chance that ordinary noise fills a window past the threshold is the tail of a binomial distribution: P(X ≥ k) where X ~ Binomial(n, p). That number, multiplied by how many windows a day your traffic produces, is how often the breaker opens for no reason.
The top row is the one worth sitting with. A five-request window is a completely ordinary setting, it looks responsive, and against a dependency with a five per cent error rate it opens four thousand times a day on nothing at all.
And the reason is not subtle once you see it: with a window of five, three unlucky failures is enough, and at five per cent error rate three failures in five requests happens about once in a thousand windows. At 200 requests a second you produce forty windows a second.
Why the window beats the threshold
Both settings move the false-trip rate, but they are not equivalent, and the difference decides how you should tune.
Raising the threshold makes the breaker less sensitive to noise and less sensitive to real failures, by roughly the same factor. You trade one error for the other along a single line.
Widening the window is asymmetric. The probability of a run of failures shrinks exponentially in the window size, while the time it takes to collect that many requests grows only linearly. So doubling the window buys orders of magnitude of quiet at the cost of a constant factor in detection. Whenever a breaker is noisy, the window is almost always the right dial.
There is a limit, and it is not statistical: a window large enough to be quiet may take longer to fill than your outage lasts. At 200 requests a second a 200-request window fills in one second, which is nothing. At 2 requests a second it takes 100 seconds, which is an eternity, and low-traffic dependencies are where breakers are genuinely hard.
That is also why most libraries have a minimum-requests setting as well as a window. resilience4j calls it minimumNumberOfCalls and explains the behaviour precisely: "at least 10 calls must be recorded, before the failure rate can be calculated. If only 9 calls have been recorded the CircuitBreaker will not transition to open even if all 9 calls have failed." Below some volume there is not enough evidence to decide anything, and the honest behaviour is to stay closed and rely on timeouts.
Its defaults are worth knowing before copying anything: slidingWindowSize 100, failureRateThreshold 50, minimumNumberOfCalls 100, permittedNumberOfCallsInHalfOpenState 10 and waitDurationInOpenState 60 seconds. A 100-request window at 50% is statistically very quiet — the problem with those defaults is not noise, it is that a low-traffic dependency never accumulates 100 calls.
Both directions of failure
Michael Nygard, who named the pattern in Release It!, framed the purpose as protecting the caller rather than the callee: the breaker exists so that a failing dependency cannot consume all of your resources waiting for it. That is a useful test for any configuration — if opening the breaker does not free a resource you were running out of, it is not doing the job the pattern was invented for.
The failure that gets noticed is the noisy breaker, because it produces incidents. The failure that does not get noticed is the breaker that has never opened, and it is at least as common — a breaker configured with a 90% threshold over 100 requests against a dependency that degrades to 60% errors will sit closed through the entire outage while every request waits for its timeout.
A practical test: look at your breaker's open/close events for the last quarter. If there are none, you do not have a working breaker, you have a dependency and some configuration. Deliberately break the dependency in a staging environment and watch whether it opens; that is the only way to know.
Work out your own settings
calculator · circuit breaker
Will your breaker trip on noise, or miss the outage?
spurious trips per day by window size, at your threshold
spurious trips per day by threshold, and what each catches in a real outage
Failures are treated as independent, which is the assumption that makes the binomial arithmetic work and the one real systems break: failures cluster, because they share a cause. Clustering makes false trips more likely than this model says, not less, so read the false-trip figures as a floor. The spurious-trip frequency also assumes windows are evaluated back to back; a breaker that slides its window continuously re-examines the same requests and trips somewhat more often. What the model gets right is the trade — widening the window cuts noise much faster than it slows detection, because the tail of a binomial shrinks exponentially in the window size while the time to fill it only grows linearly.
The defaults are deliberately a bad configuration that looks reasonable. Three things to try:
Set the window to 20. Spurious trips go from 110 a day to one every hundred days. Nothing else changes and detection slows by 0.05 seconds.
Set the baseline error rate to 0.1%. The same window that was unusable becomes fine. This is why a breaker copied from another service can behave completely differently: it is the dependency's error rate that decides, not the code.
Set requests per second to 2. A low-traffic dependency. Now the window takes a long time to fill and the breaker is slow to react no matter what you set — which is the case where a breaker is the wrong tool, and a timeout plus a retry budget is the right one.
| Symptom | Likely cause | Fix |
|---|---|---|
| Trips nobody can reproduce | Window too short for the baseline error rate | Widen the window before touching the threshold |
| Has never opened | Threshold too high, or window too long for the traffic | Test it by breaking the dependency deliberately |
| Opens correctly, then re-opens repeatedly | Too many probes on half-open | Limit concurrent probes; back off the open duration |
| Opens during deploys of the dependency | Working as designed | Shorter open duration, or exclude expected restarts |
The half-open state is where breakers fail
Two mistakes are near-universal on the way back.
Unlimited probes. When the breaker half-opens, every client that has been failing fast is now ready to try again. If they all do, the recovering dependency receives its full load at the moment it is least able to serve it, fails, and the breaker opens again. This is exactly the shape of a cache stampede: a synchronised herd arriving the instant a gate opens. The fix is the same — let a small number through, and make the rest wait or serve something stale.
A fixed open duration. If the dependency is genuinely down for an hour, a five-second open duration means every instance you run probes it every five seconds for an hour. Back off the open duration on repeated failures, the same way you would back off a retry, so the probing load falls as the outage lengthens.
And there is a question that precedes both: what does the service do while the breaker is open? "Fail immediately" is only useful if the caller has something better to do with a fast failure than with a slow one. A breaker in front of a recommendations service should return an empty list; a breaker in front of the payment provider should probably not exist, because there is no degraded answer and failing fast just loses the sale faster.
quick check
Your breaker opens several times a day and the dependency is healthy. Do you raise the threshold from 50% to 70%, or widen the window from 10 to 20 requests?
Raising the threshold makes the breaker less sensitive to noise and to real failures by roughly the same factor, so you trade one error for the other. Widening the window is asymmetric: the binomial tail shrinks exponentially in window size while the time to fill it only grows linearly. Ten to twenty requests took spurious trips from 110 a day to 0.01 here, at a cost of 0.05 seconds.
What this model gets wrong
Failures are not independent. The binomial assumes each request fails on its own coin flip. Real failures cluster, because they share a cause: one slow node, one bad deploy, one expired certificate. Clustering makes a run of failures more likely than the model says, so every false-trip number here is a floor rather than an estimate. The direction of the error is at least the safe one — real breakers are noisier than this, not quieter.
A rolling window re-examines the same requests. The trips-per-day figures assume windows are evaluated back to back. A breaker that slides its window one request at a time gets many more chances to cross the line with the same data, and trips somewhat more often than shown.
Latency failures are not counted. Most breakers count errors, and the dependency failure that matters most is usually slowness rather than errors. A dependency answering correctly in eight seconds will not trip an error-counting breaker at all, while consuming every thread you have. Breakers that count slow calls as failures handle the case that actually happens: resilience4j has slowCallDurationThreshold, defaulting to 60 seconds, which is far too long to be useful and has to be set to something near your own timeout.
Concurrency limiting may be the better tool. A breaker is binary: open or closed. An adaptive concurrency limit degrades smoothly, allowing fewer in-flight calls as latency rises rather than none at all. Netflix's concurrency-limits library applies TCP congestion-control algorithms to this, and for a dependency that gets slower rather than failing outright it is a better fit than a breaker.
The baseline error rate is not a constant. It moves with deploys, with traffic mix, and with whatever the dependency's own dependencies are doing. A breaker tuned against last quarter's baseline can become noisy without anyone changing a line of configuration, which argues for alerting on breaker state changes rather than assuming the settings stay correct.
A breaker does not fix a missing timeout. If the underlying call has no deadline, the breaker cannot protect you: threads are consumed while waiting, and the breaker opens after the damage is done. The timeout budget comes first, and the breaker is what stops you paying that timeout over and over once it is clear the dependency is down.
questions people ask
What failure threshold should a circuit breaker use?
Start from the window rather than the threshold. Work out your dependency's normal error rate, then pick a window large enough that ordinary noise almost never fills it — the binomial tail tells you exactly how often it would. Fifty per cent over 20 or more requests is a reasonable starting point for a dependency that errors a few per cent of the time.
Why does my circuit breaker keep opening when nothing is wrong?
The window is probably too short for the dependency's baseline error rate. A 10-request window at a 50% threshold against a 5% error rate opens on noise about 110 times a day at 200 requests a second. Doubling the window to 20 takes that to once every hundred days.
Should I change the window or the threshold?
The window, almost always. Raising the threshold makes the breaker less sensitive to noise and to real failures by about the same factor. Widening the window cuts noise exponentially while slowing detection only linearly, so it improves both sides of the trade at once.
Do circuit breakers work for low-traffic dependencies?
Badly. The window has to fill before anything can happen, so at a couple of requests a second a statistically sound window takes minutes. Below some volume the honest answer is a timeout and a retry budget rather than a breaker, which is why most libraries also have a minimum-requests setting.
How many requests should the half-open state allow through?
A small, fixed number — one or a few — rather than everything that has been waiting. If every client probes at once, the recovering dependency takes its full load at its weakest moment and fails again. Back off the open duration on repeated failures too, so a long outage is not probed every few seconds by every instance.
Should a circuit breaker count slow calls as failures?
Yes, where the library supports it. The dependency failure that hurts most is usually latency rather than errors: a call that eventually succeeds after eight seconds consumes a thread and will not trip an error-counting breaker at all.
The short version
Before setting a threshold, work out how often ordinary noise will fill the window. It is a binomial tail probability and the answer is often four figures a day for settings that look completely reasonable.
When a breaker is noisy, widen the window rather than raising the threshold. The window cuts false trips exponentially and slows detection linearly, so it is the only dial that improves both sides. Ten to twenty requests was four orders of magnitude here, for 0.05 seconds.
Then check the other failure: if your breaker has never opened, it is not protecting anything. Break the dependency deliberately in staging and watch. And spend the design effort on the half-open state — limited probes, backing off the open duration — because that is where breakers re-create the outage they just prevented.