Archiboard

Set R · Resilience & Scale

Retries, timeouts and the circuit breaker: Polly then, Microsoft.Extensions.Resilience now

Three retries and a two second wait is not a resilience strategy. It is a load generator with good intentions.

On a Monday morning in November 2013 a publishing platform I was responsible for returned 502 to every tenant for eleven minutes, and the cause was our own retry code. The search service behind it had slowed from 80 milliseconds to about four seconds. Our HttpClient wrapper held a Polly retry of three attempts with a fixed two second wait, no timeout and no jitter, so every request occupied a thread for the better part of twelve seconds. The thread pool starved, the load balancer probe stopped getting an answer, and Azure recycled instances that were not broken. The search service itself recovered in ninety seconds. We spent the other nine and a half minutes fighting ourselves.

That morning taught me the thing I now put at the top of every resilience sheet: a retry is a decision to send more load to a system that has just told you it is unwell. It is often the right decision. It is never a free one. The policies around it, the timeout that bounds each attempt and the breaker that stops the attempts entirely, exist to make the cost bounded rather than to make the retry clever.

The good news, twelve years on, is that you no longer write any of this yourself. In 2013 you either took the Transient Fault Handling Application Block and accepted its opinions about SQL, or you took Polly and built the plumbing around it, which in our case was a wrapper class nobody was obliged to call. Today it is Microsoft.Extensions.Http.Resilience, built on Polly v8 pipelines, attached to the client itself so there is nothing left to forget.

What the 2013 policies got wrong

Three things, and they were all mine. The first was that there was no timeout. HttpClient has a default of 100 seconds, which is not a timeout, it is a promise that the request will eventually stop. Nobody sets it lower on a shared client, so the effective per-attempt budget was whatever the slowest dependency felt like on the day. A retry wrapped around an unbounded attempt does not add resilience; it multiplies the unbounded part.

The second was fixed delays. A dozen instances that all saw the same failure at the same moment retried at the same moment, two seconds later, and again two seconds after that. We synchronised our own fleet into a metronome and pointed it at a service that was already struggling. Jitter is not a refinement, it is the difference between a retry and a denial of service you are paying for.

The third was that we retried everything, including POST. Most of our POSTs happened to be idempotent by accident, which is the worst kind of idempotent, because it stays true right up until somebody adds an audit row. We later found an endpoint that had written three identical billing rows in a single afternoon. Nobody noticed for two weeks.

The standard handler and its order

The current package chains five strategies, and the order is the interesting part. From the outside in: a concurrency rate limiter, a total request timeout that covers all attempts together, the retry, the circuit breaker, and finally a per-attempt timeout that bounds one HTTP send. Read outward and the meaning is clear. The innermost timeout says how long one try may take. The retry says how many tries. The outer timeout says how long the caller waits in total, whatever happens inside.

// .NET 9. In 2013 this was a wrapper class holding a Polly policy, and the
// call sites that went round the wrapper got none of it.
builder.Services.AddHttpClient<ArticleSearchClient>(c =>
    {
        c.BaseAddress = new Uri("https://search.internal/");
    })
    .AddStandardResilienceHandler(o =>
    {
        o.AttemptTimeout.Timeout = TimeSpan.FromSeconds(3);
        o.TotalRequestTimeout.Timeout = TimeSpan.FromSeconds(12);
        o.Retry.MaxRetryAttempts = 2;
        o.Retry.UseJitter = true;
        o.CircuitBreaker.SamplingDuration = TimeSpan.FromSeconds(30);
    });

The defaults are a 10 second attempt timeout, a 30 second total, three retries with exponential backoff and jitter, and a breaker that opens at a 10 percent failure ratio over a 30 second window once at least 100 requests have passed through it. Those numbers are tuned for a service talking to another service in the same region. For an interactive request path they are too patient, which is why the block above cuts them roughly in half. The options are validated at startup, so a total timeout shorter than the attempt timeout fails when you deploy rather than at midnight, which I appreciate more than I expected to.

The one rule I would add to the documentation is not to stack handlers. Adding the standard handler and then a hand-rolled retry handler gives you the product of the two attempt counts, and the product is always larger than the number anyone in the room intended.

The five strategies of the standard resilience handler wrapped around one outbound call your api 1. rate limiter, 1000 concurrent 2. total timeout, 12 s 3. retry, 2 attempts, jitter 4. circuit breaker, 10% / 30 s 5. attempt timeout, 3 s exactly one HTTP send search service open here and nothing below runs without jitter every instance retries in the same millisecond
Fig. 1. The pipeline reads outward. The innermost timeout bounds one attempt, the retry decides how many attempts, and the outer timeout bounds the wait for the caller regardless of what happens inside.

What you must not retry

A retry is safe when the operation is idempotent, and only then. GET is safe. PUT and DELETE are usually safe if you wrote them properly. POST is not, unless the endpoint carries an idempotency key that the server honours, and if you have not built that, assume it does not. The library gives you a one-liner for this, and I turn it on by default, then re-enable retries per client where I have actually checked the endpoints.

.AddStandardResilienceHandler(o =>
{
    // POST, PATCH, PUT, DELETE and CONNECT stop being retried.
    o.Retry.DisableForUnsafeHttpMethods();
});

The second class of things not to retry is anything that will fail identically the next time. A 400 means your payload is wrong and will still be wrong in two seconds. A 401 means you need a new token, not another attempt with the old one. A 404 means the thing is not there. The standard handler already restricts itself to 408, 429, 5xx and the transport exceptions, which is the right list, and the temptation to widen it should be resisted.

Then there is 429. A 429 is not a transient fault, it is a service telling you, politely and in writing, that you are asking too often. Retrying it on your own backoff instead of reading the Retry-After header is how a client turns a throttle into an outage. I treat a sustained 429 rate against a dependency as a capacity conversation rather than a resilience one. Sheet R-03 covers the same header from the other side, where you are the one sending it.

The storm you cause yourself

Do the arithmetic before you pick a retry count. Forty instances, each sending 25 requests per second to a dependency, is a thousand calls per second. With three retries and no breaker, a dependency that is failing everything now receives four thousand calls per second while it tries to recover. You have not added resilience, you have added a factor of four to the load at the exact moment the system can least absorb it.

The circuit breaker is what makes the arithmetic stop. Once the failure ratio crosses the threshold it opens, and calls fail immediately without touching the network, which gives the dependency room to come back. This is the part people configure last and should configure first. The default sampling window of 30 seconds with a minimum throughput of 100 requests works for a busy service; for a client that sends five requests a minute the breaker will never see enough traffic to trip, and you should either lower the throughput minimum or accept that the breaker is decoration and rely on the timeouts instead.

I also stopped treating an open circuit as an error. It is information, and the calling code should have an answer for it: last known good results from the cache, a queued job instead of a synchronous call, or a plain message that this one feature is unavailable while the rest of the product still works. A breaker with no fallback behind it converts a slow failure into a fast one and nothing more.

What I set today

For a call inside the same region on an interactive path: an attempt timeout at roughly four times the measured p99, two retries, jitter on, a total timeout equal to the user-facing budget minus what the rest of the request costs, and retries disabled for unsafe methods unless the endpoint has an idempotency key. For a background worker calling the same dependency, the same handler with a longer total timeout and more attempts, because nobody is waiting.

And one rule that has nothing to do with configuration. Every dependency gets its own named client and its own breaker. A shared pipeline across three dependencies means one sick dependency opens the circuit for the other two, and I have watched a good team lose an afternoon to that particular puzzle. Separate pipelines cost nothing, and the diagnostics they produce are worth the extra lines in Program.cs.

Drawn from