Latency rarely comes from the gateway
'The platform is slow' is the first hypothesis of every latency incident, and almost always the wrong one. The diagnostic method, and the real suspects.
It is an incident ritual: a service's latency climbs, the call goes through the platform, so "it is the gateway". The platform team spends two hours proving its innocence, the real culprit runs free in the meantime, and the scene will play again at the next incident.
The reflex has a simple explanation: the gateway is the only component every call crosses, so the only one anyone can accuse without knowing the case. The reflex also has a high cost, measured in incident hours lost on the wrong trail.
What a gateway really costs
A correctly sized gateway adds two to five milliseconds: TLS termination, token validation, the policies, the network hop. On a call that takes 300, it weighs one to two per cent.
When a latency doubles, the cause is almost always elsewhere. Here are the suspects, in the order of the frequencies we see on assignment.
The backend, obviously. The majority case and the least examined, precisely because the gateway offers a more convenient culprit. A SQL query that has stopped using its index, a GC running away, a saturated thread pool.
The connection pool to the backend. The most frequent suspect on the platform side, and the quietest: the gateway waits for an available connection before it even sends the request. The backend shows normal times, the client measures double, and the difference sleeps in a queue nobody watches.
The identity provider. Validating a token is local and fast, until it stops being so: expired key cache, synchronous introspection call, quota reached at a SaaS IdP. Typical symptom: latency in staircase steps, correlated with cache expiries.
DNS and service discovery. Resolutions that go from 1 ms to five seconds on a resolver timeout, the glibc default value, and up to thirty when several servers are unreachable. Or search domains walked through before the right name. Rare, spectacular, and invisible in every application dashboard.
A single consumer. Average latency climbs because one consumer has started sending pathological requests: pages of 10,000 items, unindexed filters. The average accuses the platform, the breakdown by consumer names the culprit in one query.
The policies that cost, and the ones that cost nothing
The gateway is rarely at fault, but it can become so: everything depends on what has been asked of it. Policies do not all carry the same price, and the gap between the cheapest and the most expensive runs to several orders of magnitude.
| Policy | Cost per call | Why |
|---|---|---|
| Routing, header rewriting | Negligible | String manipulation in memory |
| Signed token validation, keys cached | Low | One local cryptographic check |
| Local rate limiting | Low | One in memory counter |
| Distributed rate limiting | Moderate | One round trip to a shared store |
| Schema validation | Moderate to high | Grows with the size and depth of the body |
| Payload transformation | High | Parsing and rewriting the entire body |
| External call inside the policy | Very high | A third party's latency, on the path of every call |
No synchronous network call inside a policy. Querying an authorisation service, enriching from a reference system, checking a revocation remotely: each one adds its latency and its failure mode to the path of every call. If the information is needed, it gets cached with a stated lifetime.
Token validation is paid for once. The key cache must be local and refresh in the background. A platform that fetches the identity provider's keys on every call has turned a check measured in microseconds into a network round trip.
Schema validation has to be sized. It is valuable, and its cost follows the size of the body. On bulky payloads, limit it to the entry points exposed to the outside, rather than applying it everywhere on principle.
The reflex almost always missing: measure the gateway's own time policy by policy, in a test environment, before stacking them in production. Half a day of measurement saves months of suspicion.
The method: cut the call apart
Diagnosis always comes back to the same question: where does the time go? A call crossing a platform breaks down into five segments, each one measurable:
- network from client to gateway.
- gateway processing, policies included.
- wait for an outbound connection.
- backend, from the first byte sent to the last received.
- return to the client.
If your platform can give those five figures for a given call, every latency incident starts with reading a trace and finds its direction in minutes. If it cannot, diagnosis runs on mutual accusation between teams, and that is the sign that your observability stopped at volume dashboards.
The instrument is called a distributed trace, the standard is W3C Trace Context with OpenTelemetry instrumentation, and every recent gateway can propagate the context. The work is rarely technical: it consists of deciding that the five segments are measured, exported to your tooling, and kept long enough to cover an incident detected late.
The indicator to publish
A platform team that wants out of the accusation ritual is better off publishing its own weight itself, continuously: gateway traversal time, in percentiles, per API. Public, binding, with a history.
The values that make the publication usable still have to be announced, otherwise it only moves the argument. A defensible order of magnitude, to validate on your own platform before you display it: two to five milliseconds at the 50th percentile for an ordinary policy chain, around fifteen at the 99th.
The reading rule sits in the gap between the two, not in their absolute value. A 99th percentile at more than ten times the median is not the story of a slow gateway, it is the story of a queue: saturated outbound connections, runtime pauses, TLS renegotiations. That is the only reason that justifies opening the subject on the platform side.
The day latency climbs, the conversation no longer starts with "it is the platform", it starts with two figures and a gap. And the two hours of innocence to prove become two hours of diagnosis on the real trail again.
Published in November 2025.
On the same subject