When HTTP retries make failures worse.

Retries feel like the obvious resilience feature. A request fails, so the client tries again. The second attempt often works, the error disappears, and the policy earns its place in the shared platform package. The danger is that the word “retry” describes a mechanism, not a safety guarantee.

A retry changes the amount of traffic, the time a caller waits and the number of times an operation may happen. Those are architectural effects. If they are not deliberate, the resilience pipeline can become the thing that makes the incident larger.

The retry storm

Imagine a downstream service beginning to time out under load. Every caller waits, fails and immediately sends two more requests. The struggling service now receives three times the work while its own threads and connections are already occupied. If several services share the same fixed delay, their next attempts arrive together as another wave.

Exponential back-off and jitter help because they spread those attempts over time. They do not create capacity. A sound policy still needs a finite attempt count, a total timeout and a decision about which failures are genuinely transient. Honour Retry-After where the server provides it; it knows more about its recovery window than the caller does.

The unsafe method problem

A GET can normally be repeated without changing state. A POST might charge a card, create an order or publish an event. The first request can succeed while its response is lost. Retrying then repeats the side effect even though the client only observed a timeout.

services.AddHttpClient<PaymentsClient>()
    .AddStandardResilienceHandler();

await client.PostAsync("/payments", content, cancellationToken);

The durable answer is application-level idempotency: a stable request key that lets the server recognise and return the result of the first operation. Where that contract does not exist, disable retries for unsafe HTTP methods rather than hoping a timeout means the server did nothing.

services.AddHttpClient<PaymentsClient>()
    .AddStandardResilienceHandler(options =>
    {
        options.Retry.DisableForUnsafeHttpMethods();
    });

Latency is part of correctness

Three attempts with a ten-second timeout are not a ten-second operation. They are a queue of waits that may outlive the user request, the caller's cancellation budget or an upstream load balancer. Resilience policies should fit inside one end-to-end deadline. Cancellation must flow through every attempt so abandoned work does not continue consuming connections.

• Set a total timeout, not only a timeout per attempt.
• Propagate the caller's CancellationToken.
• Limit concurrency as well as attempts.
• Measure retries separately from successful first attempts.

Make the policy reviewable

The risky shapes are visible in source: an unsafe method behind a generic retry handler, a factory-created client held for too long, a response stream without a clear owner, or an available cancellation token omitted at the call site. Those are good candidates for compile-time feedback because the warning can appear while the policy is still being written.

HttpClient.Resilience.Analyzers packages those checks as focused Roslyn diagnostics and code fixes. The point is not to ban retries. It is to require the code to state why repetition is safe, who owns the response and when the caller is allowed to stop waiting.

Resilience is controlled failure, not repeated optimism.

← Back to writing