Timeouts, retries, and not amplifying an outage

Everything so far is about code that reads well. This section is about code that is still answering when something it depends on is not.

Every remote call has a timeout, and you chose it

A call with no timeout inherits one from a layer you have not read — an HTTP client default, a socket setting, sometimes nothing at all. Under load, that is the difference between a slow dependency and a thread pool fully occupied waiting on it. The failure does not stay where it started: your service stops answering requests that have nothing to do with the slow dependency.

Two numbers, not one:

They fail for different reasons and deserve different values. A connect timeout of 30 seconds is almost always wrong; if the TCP handshake has not completed in a second or two, it is not going to.

And the timeout must be shorter than the caller’s patience. A chain where each hop allows 30 seconds means the browser gave up long before the innermost call did, and every thread in between is still holding on for work nobody wants.

Retries amplify outages

A retry is a second request to a service that just failed. Three retries across four instances is sixteen requests arriving at something already struggling — which is how a partial degradation becomes a full outage.

Three rules make a retry safe:

Only on errors that could plausibly succeed next time. A timeout, a connection reset, a 503: yes. A 400 or a 422: never — the request is wrong and will be wrong again. Retrying a validation failure is pure load with a guaranteed outcome.

Never on a non-idempotent write without an idempotency key. “Create payment” retried after a timeout may well create two payments, because a timeout tells you nothing about whether the other side processed it. The key lets the server recognise the repeat and return the first result.

With backoff and jitter. Fixed-delay retries synchronise: every client that failed at the same moment returns at the same moment, and the load arrives in waves that keep the dependency down. Exponential backoff spreads the waves out; jitter — randomising each delay — is what breaks the synchronisation, and it matters more than the exponent. Without it, a thousand clients backing off identically still arrive together, just later.

Aspect order is not cosmetic

Resilience4j’s default order wraps Retry outside CircuitBreaker:

Retry ( CircuitBreaker ( TimeLimiter ( Bulkhead ( call ) ) ) )

Read that from the inside out and the consequence is concrete: each retry attempt is a separate call through the breaker, so failed attempts count towards opening it — and once it is open, the remaining retries fail immediately without touching the network. That is usually what you want: the retries stop being load the moment the breaker decides the dependency is unwell.

Invert it — breaker outside retry — and one logical call becomes one breaker event no matter how many attempts it made, so the breaker sees a third of the failures and takes three times as long to open.

Failing fast is a feature

The instinct when a dependency is slow is to wait longer. It is backwards. Every request waiting is a thread, a connection, and a slot in a queue that a healthy part of the system could have used. A fast failure with a degraded answer — a cached value, a partial page, an explicit “try again shortly” — keeps the rest of the service usable.

That is also why enlarging a connection pool to fix latency usually makes things worse: more concurrent queries against a database that is already the bottleneck means more contention, longer queries, and now the pool is exhausted too.

The configuration in code, with values that have been through a production incident: Resilience4j Patterns.