Your Retry Budget Is Longer Than Your Users' Patience

An internal dashboard of mine fails sometimes. Not always, which is the annoying part. You open it, and instead of the numbers you get the error panel, and then you refresh and it works.

Intermittent failures reward patience over cleverness, so I went looking properly. Almost everything I suspected was wrong, and the thing that was actually broken had been sitting in a config file for months looking like a safety feature.

Everything obvious was wrong

I had five theories. I measured all five and threw all five away.

Data volume. The table behind the page holds fewer than twenty-five rows. Asking for twenty-five, a hundred or two hundred returns a byte-identical response. There is no size problem here and there never was.

A slow query, or the database being throttled. During the failure the database was running at zero to three percent utilisation, with the CPU flat at zero. It was not struggling. It was asleep.

Serverless auto-pause. A reasonable guess, except production is not serverless. It runs on small provisioned instances, which matters more than it sounds and I will come back to it.

Connection exhaustion at the network layer. Zero failed, zero pending. Nothing there.

Cold start in the runtime. This one I actually believed. It measured at 1.25 seconds of difference between cold and warm. The symptom was twenty-something seconds, so it is an order of magnitude too small to matter.

My own tooling faked a cold start

Before the real answer, a detour that cost me two wrong diagnoses.

I was timing requests through a tunnel into the cluster. The first request through a fresh tunnel is about a second slower than the ones after it, and that has nothing to do with the service being called. It is the tunnel setting itself up.

So I measured, saw the first call was slower, and concluded I had found a cold start. Twice. The fix is unglamorous: throw a few junk requests at it first to warm the tunnel, then start timing. If you are benchmarking through any kind of proxy or forwarder, your first number is about your tooling, not your system.

Then the number that explained it

I pulled a week of connection telemetry. Seven days is 182 hourly buckets. The database accepted connections in two of them.

Twenty-five successful connections in a week, all clustered in two hours. One hundred and eighty hours with none at all. Nothing touches this database on a schedule. It is only ever woken up by a person opening the page.

Which means the connection pool is never warm. There is no steady trickle keeping a connection alive, so every single visit opens a brand new one. The first user is always the first user.

And in that same week there were exactly two connection failures, both inside the hour of the incident I was chasing. Two transient blips, in a system where nothing else was ever connecting.

The budget nobody added up

Here is the part I would want someone to tell me.

The connection timeout has a sixty second floor. On top of that the data layer retries up to five times. The browser making the request gives up at thirty seconds.

So a transient blip lasting two seconds gets handed to a retry policy that is permitted to spend double the caller's entire patience before it finishes. The observed failures ran between eighteen and twenty-seven seconds, which is already brushing the limit, and the configuration allows far worse.

That is not resilience. The retries do eventually succeed, and the success arrives at an address nobody is standing at any more. All the policy bought was a slower, more expensive way to show the same error.

A retry budget is only a safety feature if its worst case fits inside the deadline of whoever is waiting. Mine had never been checked against that number. The two values live in different repositories and were chosen years apart by me, for reasons that were locally sensible on both occasions.

Where the sixty seconds came from

The floor is not arbitrary. Serverless databases pause when idle and can take twenty to forty seconds to wake up, so a short connection timeout would fail every first request. Sixty seconds is a defensible number for that situation.

Production is not that situation. It is provisioned, it never pauses, and it does not need waking. The floor protects against a problem this system does not have, while allowing a doomed connection to hang long past the point anyone is watching.

Inherited safety settings are hard to see precisely because they look responsible. Nobody audits a generous timeout. It reads as caution rather than as a decision that might not apply any more.

What I am changing

Not fixed yet, so I will describe it as intent rather than achievement.

Drop the sixty second floor where the database is provisioned, and let a bad connection fail quickly instead of hanging. Bound the retry budget so the worst case lands comfortably under the caller's deadline, which means roughly three retries with a short backoff rather than five with a long one. And put something on a schedule that touches the database every few minutes, which keeps a connection alive and has the useful side effect of noticing trouble before a person does.

The last one is the one I find most interesting, because the root cause is really that this system is used too rarely to stay warm. Rarely-used things are permanently cold, and the cost lands entirely on whoever shows up first.