Resilience

The most common failure mode of a distributed system is not the outright outage, it is slowness that spreads: a degraded service holds on to its callers, who fill their queues and hold on to theirs. Four mechanisms work against it. Applying them at the gateway belongs to the mediation pillar. Designing them is a matter of architecture, and the settings below are the heart of it.

Timeouts

Every outgoing call carries a time limit. Without it the other mechanisms are useless: a call that waits indefinitely holds a resource indefinitely.

The tuning rule: decreasing from the outside in. If the caller allows two seconds in total, the internal call does not get three. Each tier waits less than the one above it, otherwise the waits stack up instead of failing cleanly, and the caller gives up while the chain keeps working for nobody.

Retries

A retry is justified only if three conditions hold: the operation is idempotent, the error is one that can go away, a timeout or a 503, never a 400, and the total volume is bounded. Retrying a creation produces duplicates, retrying without a bound turns an incident into an unintentional attack on your own backend.

The settings that hold: one or two retries at most, spaced out with some randomness so that clients do not all come back at the same instant. Plus a global budget, the share of traffic allowed to be retries, beyond which you stop. On the API side, providing an idempotency key in the contract makes retries safe, creations included.

The circuit breaker

The circuit breaker stops calling a service that is failing, for a set period, then tries again cautiously. Three states: closed, traffic passes and failures are counted over a sliding window. Open, calls are refused immediately without touching the service. Half-open, a few trial calls decide what comes next.

The only hard decision is not the mechanism, it is the answer given while the circuit is open: an immediate 503, a cached value you stand behind, an explicit partial response. That choice is a business one, yesterday's balance suits a retail bank and not a trading floor, and it is taken at design time, not in the configuration during the incident.

Bulkheading

Bulkheading isolates resources by destination or by consumer: separate connection pools, distinct queues, so that one slow destination does not exhaust the shared resources. At the gateway it takes the form of per consumer limits: without them a single client in a loop degrades the service for everyone else, which turns a bug at one partner into a platform incident.

Where all this sits, and how to check it

The caller protects itself, the callee cannot do it on its behalf: circuit breaker and timeouts live on the calling side. The gateway carries them for each backend and so protects every consumer, services carry them between themselves, and the service mesh can supply timeouts and retries without touching the code, which makes the settings uniform.

None of these mechanisms is set once and for all: thresholds calibrated for ten times less traffic open too early or too late. And none of them is taken on trust: actually cutting off a backend in a test environment, and watching what consumers receive, is the only test that counts. A circuit breaker never triggered before its first incident is discovered during the incident.