When work runs twice

Delivery exactly once is not a property anything can offer end to end, and a vendor advertising it is describing a combination, not a guarantee: at-least-once delivery plus handling at the receiving end that is safe to repeat. The second half is the part that has to be built, and it is the part left out when the phrase is taken at face value.

The mechanism is a repeat key: a stable identifier derived from the event itself, not generated at the moment of sending. The same event yields the same key on every attempt. A key generated per attempt is a fresh key on every retry, which is indistinguishable from having no key at all, and it is a mistake that looks correct in every test where nothing is retried.

The receiving side records the keys it has already processed and short-circuits on a repeat, returning the original outcome instead of doing the work again. How long those records are retained is a decision, not a default: long enough to cover the longest retry window the upstream system will ever use, which means reading the vendor's documentation instead of estimating.

The check and the write have to be atomic, and this is the subtlety that defeats most first attempts. Two copies of the same event arriving at the same moment will both consult the record of processed keys, both find nothing, and both proceed — the deduplication logic is present, correct in isolation, and completely ineffective. The fix is a uniqueness constraint in the database doing the work, so that the second write fails on the constraint instead of on a check that has already passed. Deduplication implemented as a read followed by a write is a race condition with good intentions.

Natural keys are preferred over synthetic ones wherever the domain supplies them — an invoice number, an order identifier, the event id the sending system already assigned. A key the business would recognize is also a key somebody can search for during an incident, which is when it will be needed.

Where the operation is a write into a third-party system, that system's own protection against repeats is used if it exists. Payment providers offer it precisely because this is where a duplicate is most expensive, and declining to use it in favor of a local check means the guarantee stops at the network boundary — exactly where the retries originate.

Ordering is not guaranteed and is never assumed. Events arrive out of sequence routinely, and a handler written on the assumption that an update follows the creation it depends on will pass every test and fail in production the first time the network is slow. Handlers either tolerate arriving early, holding the event until its predecessor lands, or the system establishes order explicitly with a version or sequence number.

Stale updates are rejected, not applied. Comparing a version or a timestamp before writing is what prevents an older value from overwriting a newer one simply because it happened to arrive second, which is the specific way last-write-wins misbehaves under out-of-order delivery, and it produces data corruption that no error will ever describe.

Retries back off instead of repeating immediately, and the backoff has jitter so that many clients recovering from the same outage do not synchronize into a second one. A retry storm against a service that is already struggling is the standard mechanism by which a brief degradation becomes a sustained outage.

Some operations cannot be made safe to repeat because their effect leaves the system entirely — sending an email, charging a card, dispatching a physical item. For these, a record is written before the attempt and reconciled after it, so that an ambiguous outcome becomes a duplicate somebody can find and resolve rather than one indistinguishable from a legitimate second transaction.

Anything that exhausts its retries lands in a durable holding area with the failure reason attached, and that destination is worked rather than accumulated. A holding area nobody empties is a record of everything the business has silently failed to do, which is useful evidence and no substitute for having done it.

Testing includes replaying the same event repeatedly, delivering events out of order, and delivering one after a long delay. All three are cheap to simulate, all three happen in production regularly, and none of them appear in the test suites of most systems that depend on getting them right. This is where the defects are.

Reconciliation runs behind all of it as the backstop, because every mechanism above can be defeated by something nobody anticipated. Periodically comparing counts and totals against the system of record catches the duplicate that slipped through, the event that was dropped rather than duplicated, and the window during an incident when the keys were not being recorded at all. Safe repetition prevents the ordinary failure; reconciliation is how the extraordinary one is found before a customer finds it.

The cost is worth stating instead of glossing. Idempotent handling means a durable record of processed keys, a lookup on the path of every operation, and a retention policy somebody has to own. For a system moving a handful of events an hour that overhead is invisible; for one moving a great many it is a genuine design constraint that shapes the storage underneath. It is still cheaper than the alternative, and the alternative is not 'no overhead' but 'overhead paid during an incident'.

None of this is interesting work and all of it is the difference between an integration that can be operated and one that produces incidents nobody can reconstruct. A customer who receives two identical invoices does not experience an at-least-once delivery guarantee. They experience a company that cannot count, and they are not wrong.