Idempotency in Distributed Systems

There is one sentence that separates experienced backend teams from inexperienced ones: "This message can arrive twice." Anyone who runs distributed systems long enough stops wishing duplicates away and starts building for them. Because the duplicate is not a bug you eliminate — it is an unavoidable consequence of messages traveling over a network that can swallow responses at any moment.

The only reliable answer is idempotency: an operation is idempotent if executing it multiple times produces the same state as executing it once. If a charge is idempotent, a "pay" command delivered twice does not cause a second charge. This text explains why duplicates inevitably occur, why "exactly-once" is an illusion, and how to use idempotency keys, HTTP semantics, outbox and inbox to build systems that tolerate duplicate delivery and repeated execution — independent of framework, queue and cloud. The tools differ from stack to stack; the principle does not.

Why duplicates occur

The root of all duplicates is a single, unsolvable problem: when a sender sends a request and gets no response, it cannot know what happened. Three cases look identical to it — the request never arrived; it arrived and was processed, but the response was lost; it is still in transit. All the sender sees is: no response. And the only action that fixes the first case is a retry — which, in the second case, produces a duplicate.

Diagram: the server cannot tell whether the first request is new or a retry — and without idempotency it charges twice.

This constellation occurs everywhere: a client timeout triggers a retry. A load balancer re-forwards a request after a brief outage. A queue delivers a message a second time because the consumer crashed before it could acknowledge it. A network hiccup repeats a packet. None of these cases is a bug in the software — they are the normal operation of a distributed system. Duplicates are not the exception you react to, but the rule you build for.

Exactly-once vs. at-least-once

There are three conceivable guarantees for message delivery, and understanding them shapes the entire design.

At-most-once delivers at most once: no duplicate, but possible loss — usable only where a lost event does not matter. At-least-once delivers at least once: no loss, but possible duplicates — the honest normal case for every serious queue and every retry mechanism. Exactly-once delivery is impossible in the general case, because delivery and acknowledgment cannot be combined atomically over a network: something can always crash between "processed" and "acknowledged."

What systems call "exactly-once" is therefore almost never exactly-once delivery, but at-least-once delivery plus idempotent processing — together, that yields "effectively-once": the message may arrive multiple times, but its effect occurs exactly once. That is the only achievable state, and the only one you need.

The reason exactly-once delivery is impossible is the same lost ACK as before, just one level up: the broker would have to deliver the message and receive the consumer's acknowledgment in the same, indivisible moment — but between the two lies a network that can swallow the acknowledgment. However cleverly you twist it, a window remains in which the broker does not know whether the consumer got the message. It can only choose: when in doubt, redeliver (at-least-once), or when in doubt, don't (at-most-once). An "exactly once" without this window does not exist — only the choice of which side of the doubt you would rather err on.

The recommendation: design for at-least-once and make processing idempotent; treat promises of "exactly-once delivery" with suspicion. The price is that every state-changing process must carry its own dedup logic instead of relying on the infrastructure. We decide differently for data whose loss has no consequences — non-critical telemetry, for example; there, at-most-once is enough, and you can skip idempotency entirely.

Idempotency keys

The most general way to make an inherently non-idempotent operation idempotent is the idempotency key: a unique key that the sender generates per logical operation and sends along. The server keeps a store that maps keys to results. When a key arrives for the first time, the operation is executed and the result is stored under that key. When the same key arrives again, the server returns the stored result without executing the operation again.

Diagram: the same key returns the same result the second time — without triggering the effect a second time.

Three details determine correctness. The key must identify the logical operation, not the technical request — for a payment, the payment transaction, not the HTTP call. Besides "done, result X," the store also needs an "in progress" state; otherwise two duplicates arriving at the same time will both execute the operation before either has stored anything. And the retention period must be at least as long as the largest retry window; otherwise the key expires before the last retry arrives.

The recommendation: give state-changing or money-moving commands a sender-generated idempotency key and a store that also knows the "in progress" state. The price is an additional store with its own consistency and cleanup obligations, plus coordinating the key across system boundaries. We decide differently for operations that are idempotent by nature — more on that shortly; there, the key is superfluous machinery.

HTTP idempotency

The HTTP protocol already carries a notion of idempotency, and using it correctly saves you from building your own machinery.

GET, PUT and DELETE are idempotent per the specification: a PUT that sets a state can be sent any number of times, and the result stays the same; a DELETE deletes nothing the second time. POST and PATCH are not — a POST that creates a resource or triggers a payment creates a second one the second time. Exactly these need an idempotency key.

One caveat matters: idempotent by method does not mean idempotent in implementation. A PUT that appends internally instead of setting is not idempotent despite its method. HTTP semantics are a promise you have to keep, not an automatism.

The recommendation: choose idempotent methods where the semantics allow it — preferring set over append — and require an idempotency key for non-idempotent operations. The price is that you sometimes have to phrase operations more awkwardly as "set" instead of conveniently incrementing. We decide differently when the business meaning is inherently non-idempotent — for example "add an entry to a log"; then the key carries the responsibility, not the method.

Queue processing

Message queues almost always deliver at-least-once. The reason is the same lost ACK: a consumer processes a message but crashes before acknowledging it — so the queue delivers it again. Expired visibility windows and rebalancing also lead to redelivery. The consequence is inescapable: a consumer must never rely on single delivery; it must be idempotent.

Idempotency in the consumer can come from several sources:

The order of effect and acknowledgment is also central: you acknowledge a message only after its effect has been durably committed. If you acknowledge first and write afterwards, a crash in between loses the message.

The recommendation: treat every consumer as at-least-once and safeguard its effect through one of the sources above; acknowledge only after the durable commit. The price is a dedup mechanism per consumer and the care needed to order effect and acknowledgment correctly. We decide differently for naturally idempotent handlers — for example one that merely sets a value to the state contained in the message; there, no additional dedup layer is needed.

Payments

No area shows the cost of missing idempotency as relentlessly as payments. A double charge is not a silent data error but visible damage to the customer, a support case and a breach of trust. This is exactly where the lost ACK is most dangerous: the "pay" command is the operation you can least afford to execute twice, and the one most likely to be retried.

The safeguard runs through the entire chain. The client generates an idempotency key per payment transaction — not per click. Your own service deduplicates on it, so that a retried request does not trigger a second payment. And the payment provider itself accepts an idempotency key and deduplicates on its side. Each stage relies on the next one respecting the key; no stage relies on it "probably not happening twice."

The recommendation: carry one idempotency key per payment transaction through the entire chain — client, your own service, provider — and make the "pay" command strictly idempotent. The price is coordination across system boundaries and a key that must be cleanly tied to the payment transaction, not to the technical request. We never decide differently: with money, there is no counter-case in which we do without idempotency.

Retries

Retries are the mechanism that turns at-least-once into reality — and at the same time the main source of duplicates. Without retries, you lose messages on every transient error; with retries, you create duplicates. That is not a contradiction but the reason retries and idempotency belong together: you can only safely retry what is idempotent. Retrying a non-idempotent operation is exactly how a double charge comes about.

Safe retries need more than a try again. They need exponential backoff with jitter, so that not all clients strike again at the same moment and trigger a retry storm. They need an upper limit, so that a permanently broken operation does not circle forever. They must distinguish retryable errors — timeout, network error, temporary overload — from non-retryable ones, because a rejected input does not become valid by retrying it. And they need an endpoint for what has been given up on, a dead-letter store, instead of losing it silently.

The recommendation: retry only idempotent operations, with backoff, jitter, an upper limit and a dead-letter path. The price is additional logic and a place where failed messages sit and have to be monitored. We decide differently for an operation that cannot be made idempotent and whose repetition would cause damage; then you do not retry automatically but escalate the error to a human or to a deliberately built compensating step.

Outbox pattern

A particularly treacherous problem arises at the boundary between the database and the message broker: the dual-write problem. An operation is supposed to change a state and publish an event — but there is no shared transaction spanning database and broker. If you write the database first and publish afterwards, the event is lost if a crash happens in between. If you publish first and write afterwards, the event is a lie if the write fails.

The outbox pattern solves this by bringing both writes into one transaction. The event is not published directly but written to an outbox table in the same database transaction as the state change — atomically, so either both or neither. A separate relay process reads the outbox, publishes the events to the broker, and then marks them as sent.

Diagram: state and event are written atomically; the relay delivers at-least-once, the consumer deduplicates.

Because the relay can crash between publishing and marking, delivery is once again at-least-once — the event can land at the broker twice. That is fine as long as the receivers are idempotent. The outbox guarantees that no event is lost and none is sent that did not actually happen; the duplicates are caught on the other side.

The recommendation: handle every combined write of state and event through an outbox in the same transaction instead of publishing directly. The price is an additional table, a relay process and the duplicates it produces. We decide differently when no event leaves the system — when the state change and its effect live in the same database; then there is no dual write and no need for an outbox.

Inbox pattern

The outbox protects the sender side; the inbox protects the receiver side and is its mirror image. To reliably detect a duplicate, the consumer stores the ID of every processed message in an inbox table — in the same transaction as the effect. If it finds an ID already present, it skips the message.

The decisive point is the shared transaction. Because the dedup entry and the effect are committed atomically together, there is no state in which the effect has occurred but the ID has not been stored — or vice versa. This is exactly what separates correct dedup from apparent dedup: if you check the ID in one transaction and apply the effect in another, a crash in between can decouple the two, and a duplicate slips through or a message is lost.

Together, outbox and inbox form the end-to-end chain: the sender loses nothing (outbox), the receiver applies the effect exactly once (inbox), and in between, delivery is free to duplicate. That is "effectively-once" across the entire message path.

The recommendation: deduplicate in the consumer via an inbox table whose entry is committed in the same transaction as the effect. The price is another table with cleanup obligations and a transaction that encloses both dedup and effect. We decide differently when the effect itself is already idempotent — a unique constraint or a conditional update; then the database handles the dedup, and a separate inbox would be redundant.

Common mistakes

The recurring patterns in which idempotency fails:

• Trusting "exactly-once" delivery, which does not exist, instead of assuming at-least-once.
• Consumers that assume single delivery and carry no dedup.
• Automatically retrying non-idempotent operations — the direct route to a double charge.
• Dedup check and effect in separate transactions, so that a crash in between decouples them.
• Storing an idempotency key without expiry (unbounded storage) or with too short an expiry (window smaller than the retry).
• Choosing a non-unique key — a timestamp, a guessed ID — that allows collisions or gaps.
• Acknowledging before the effect has been durably committed — the message is lost in a crash.
• Writing state and event separately without an outbox — lost or fabricated events.
• Forgetting the "in progress" state, so that two simultaneous duplicates are both processed.
• Relying on message order for correctness, which at-least-once does not guarantee.

Decision checklist

Questions to ask before any state-changing processing. They are diagnostic questions, not verdicts.

• Does this operation change state or move money? If so, it must be idempotent — without exception where money is involved.
• Is it idempotent by nature, or does it need keys or dedup? Natural idempotency is the cheapest.
• Is there a unique key per logical operation, generated by the sender? Not per technical request.
• Are dedup and effect committed in the same transaction? If they are separate, it is not real dedup.
• Do we acknowledge only after the durable commit of the effect? Otherwise, loss looms.
• Does the key's retention cover the largest retry window? Otherwise it expires too early.
• Do we handle simultaneous duplicates, not just consecutive ones? That requires an "in progress" state.
• Are we crossing the database/broker boundary? If so: outbox and inbox.
• Are retries bounded, with backoff, jitter and a dead-letter path? Otherwise storms and silent loss loom.

FAQ

Don't modern brokers offer exactly-once? What brokers call that is usually effectively-once within their own boundary — dedup in the broker plus transactional acknowledgment. As soon as a message triggers an effect outside the broker, such as a charge, at-least-once applies again, and idempotency is your responsibility. The guarantee ends at the broker's boundary.

Isn't a unique constraint enough? Often, yes — a unique constraint is a form of idempotency: the second insert attempt fails. What matters is that the key represents the logical operation and that you handle the collision case cleanly, i.e. treat "already exists" as success with the existing result, not as an error.

How long do we keep idempotency keys? At least as long as the largest window in which a retry can arrive — including retries, redeliveries and manual repeats. After that, the key may expire. Too short, and a late retry is treated as new; unbounded, and the store grows forever.

What if two duplicates arrive at the same time? A pure result store is not enough then, because both start before either has stored anything. You need an "in progress" state or a lock on the key: the first duplicate claims it, the second waits or is rejected. Only that makes idempotency correct under concurrency as well.

Does every operation need idempotency? No. Pure reads do not need it, and naturally idempotent operations — setting a value, deleting by ID — already carry it within them. The effort applies to state-changing and money-moving operations that are not already idempotent on their own. Anything else would be machinery without return.

Outbox or a broker transaction? The outbox works everywhere, without special broker capabilities, because it only needs a database transaction. If the broker offers a genuine transactional coupling to the database, you can weigh it up — but the outbox is the portable, framework- and cloud-neutral default.

Further reading

• Idempotency — the compact definition of the term.
• Queue or synchronous processing and Introducing asynchronous processing — when queues make sense and how to introduce them.
• API versioning without breaking customers — idempotency keys are part of a robust API contract.
• Zero-downtime database migrations — the same discipline of repeatability, applied to backfills.

It is grounded in the Batunet Engineering Method: design for failure, build in small reversible steps, observability from day one.

A distributed system that must never deliver twice is a system you cannot build. The honest goal is not to prevent duplicates, but to make them so inconsequential that no one ever notices them.