The retry that sent the invoice four times

Retries feel like unambiguous good engineering. Networks are unreliable, so try again. Every HTTP client ships with a retry policy, every queue redelivers, every agent framework retries failed tool calls.

Then a customer receives the same invoice four times, and the post-mortem finds no bug — just three layers of retry, each behaving exactly as designed.

Retries multiply

The multiplication is the part people miss. Retries compose, and they compose multiplicatively.

An agent framework retries a failed tool call three times. The tool's HTTP client retries three times on timeout. The job queue redelivers the whole run on worker failure. One logical operation now has up to twenty-seven attempts, and nobody chose that number — it's the product of three independent decisions made by three different people, in three different layers, each of which looked reasonable in isolation.

Add an agent that decides on its own to try a different approach after an ambiguous result, and the count stops being bounded by configuration at all.

The first rule: retry at exactly one layer. Pick it deliberately — usually the outermost one that can see whether the operation as a whole succeeded — and turn retries off everywhere else. Defaults are on almost everywhere, so this means actively disabling them.

Timeouts are not failures

The dangerous case isn't a clean error. A 500 means it didn't work; retrying is fine.

A timeout means you don't know. The request may have been received, processed, and the response lost on the way back. From the client's perspective these are identical, and the difference determines whether a retry is a fix or a duplicate.

This is why the naive policy — retry on any non-2xx and on timeout — is unsafe for non-idempotent operations. The very case where retrying is most tempting is the case where you have the least information.

Three ways out, in order of preference:

Idempotency keys. Send a key with the request; the provider returns the original response for a repeat. Where supported, this collapses the problem entirely.
Query before retry. After a timeout, ask whether the operation landed. Requires an endpoint that can tell you, and a natural key to look it up by.
Don't retry, escalate. For operations that are irreversible and unverifiable, a failed attempt should surface to a human rather than repeat automatically. Sending nothing is usually better than sending twice.

Classify the operation first

Before configuring any retry policy, sort the operation:

Naturally idempotent. Reads. set_status(closed). PUT to a known key. Retry freely.

Idempotent with a key. Payments, message sends, record creation at providers that support keys. Retry with the key.

Not idempotent, verifiable. Creating a record with a natural key you can search for. Retry after checking.

Not idempotent, not verifiable. Sending email through a provider with no dedupe and no search. Do not retry automatically. This category is smaller than people assume once they look, and it's where duplicates come from.

Backoff, jitter, and the thundering herd

Immediate retries against a struggling service make it worse. Exponential backoff is standard, and the part frequently omitted is jitter — randomisation of the delay.

Without it, a thousand clients that failed simultaneously all retry simultaneously, then back off in lockstep and retry simultaneously again. The synchronised load keeps the service down. Full jitter — a random delay between zero and the computed backoff — spreads it.

Also cap the total: a bounded number of attempts and a deadline. An operation retrying for six hours has usually stopped being useful, and it's occupying a worker the whole time.

Circuit breakers

Retrying against a service that is comprehensively down is waste that hurts both sides. A circuit breaker tracks failure rate and stops trying after a threshold, periodically letting one request through to test recovery.

The operational value is mostly in what it makes visible: an open circuit is an unambiguous signal about which dependency is broken, and it's far clearer than a slow accumulation of timeouts across a dozen workers.

Retries in agentic workflows

Agents add a failure mode conventional systems don't have: the model retries on its own initiative, because it judged the result unsatisfactory rather than because anything failed.

This is invisible to your retry configuration. The tool call succeeded, returned 200, and the agent called it again with slightly different arguments because it didn't like the answer. No retry counter incremented anywhere.

Two defences:

Per-run caps on side-effecting calls, enforced by the runtime rather than requested in the prompt. A workflow that should send one email gets a budget of one.
Make repeated calls visibly identical. If the second call carries the same idempotency key, the provider absorbs it. If it's semantically the same but syntactically different, it won't.

The check to run

Take your most critical workflow and answer: if every step ran twice, what would a customer notice?

If the answer is "nothing," the retry policy is safe. If it includes anything a person would complain about, that step needs an idempotency key, a gate, or a rule against automatic retries — before the network makes the decision for you.