Building Levee: A Self-Tuning Circuit Breaker
The View From the Hot Seat
Circuit breakers are wonderful intermediaries between two high-traffic synchronous network services. They prevent the caller from overloading the callee when the latter is in trouble. In turn, this protects the caller from getting backed up with too many pending requests. This is all assuming that the circuit breaker is well configured, as are the timeouts and rate limits.
Unfortunately, I’ve seen way too many incidents where the “well configured” assumption is not held true. Too often, concurrency is allowed to bloat until it exhausts resources and that in turn happens because equally often, the timeout is set as if it were a prayer instead of a protective limit.