Idempotency: making every retry safe in a distributed system

A payment request times out after ten seconds. The client has no idea what happened at the other end. Maybe the request never arrived. Maybe it arrived, the card was charged, the order was written, and the reply died somewhere on the way back. From where the client sits, those two outcomes look identical.

The client has to pick one. If it gives up, it may be abandoning a customer who has already paid. If it retries, it may be charging them twice. Neither choice is safe until the server can tell that the second request is the same request as the first.

That is all idempotency is: giving a request an identity, so the server can recognise it coming back. Every external call in PostEngage is a job with retries, dead-lettering and alerts, which means retries are the default there. Duplicates are the default too, unless the thing being called can spot them.

The failure you cannot see from the client

There are three moments a call can break, and Stripe's Brandur Leach lists them plainly: "The initial connection could fail as the client tries to connect to a server", "the call could fail midway while the server is fulfilling the operation, leaving the work in limbo", and "the call could succeed, but the connection break before the server can tell its client about it".

Only the first is harmless. The third is the expensive one, and it is invisible from outside. A gateway timeout, a load balancer with a 30-second idle limit, a mobile connection that drops as the user walks into a lift: all of them produce a client that did not hear an answer and a server that did the work.

The AWS Builders' Library puts the same case in one sentence in Making retries safe with idempotent APIs: "it would be undesirable for the EC2 instance launch workflow to retry a failed call to create an EBS volume and end up with two EBS volumes". Swap volumes for charges, invoices or WhatsApp messages and you have most of the duplicate bugs I have been asked to look at.

What idempotent actually means

RFC 9110 defines it: "A request method is considered 'idempotent' if the intended effect on the server of multiple identical requests with that method is the same as the effect for a single such request." GET, HEAD, PUT, DELETE, OPTIONS and TRACE are idempotent by definition. POST is not, which is why creating things is where this bites.

Two details in that sentence matter.

It says effect, not response. A second DELETE that answers 404 is still idempotent, because the resource is still gone. Your replayed order can return the same 201 and the same order id; it does not have to be byte-identical, it has to leave the system in the same state.

And it says intended. Nothing makes a method idempotent for free. The same spec is clear that idempotency is what allows a client to retry at all: "Requests that are idempotent MAY be automatically retried by a client in the event of an underlying connection failure." Automatic retry is a feature you unlock by doing this work, not a default you get.

Count your own duplicates

The arithmetic is small enough to do on the back of an envelope. Take the requests that mutate something, multiply by the share that time out, and multiply by the share of those that had actually succeeded on the server. That is roughly how many duplicates you create, per day, for ever.

At 20,000 order requests a day, with 0.2% of them timing out and half of those having already worked, that is about 20 duplicates a day. Six hundred a month. At a $40 average order, roughly $24,000 a month of charges to find and refund, one customer at a time.

Two things surprise people the first time they run this.

The first is that the retry policy is not the lever. Going from three retries to one barely changes the number of duplicates, because almost all of them come from the first retry. Going to zero retries removes them, and replaces them with something worse: at those inputs, 1,200 customers a month who get a timeout and no answer, about 600 of whom were charged anyway.

The second is how cheap the fix is. The key store for that traffic, with a 24-hour window and a kilobyte per key, is about 20 MB. You are choosing between 20 MB of Redis or Postgres and $24,000 a month of refunds.

The contract: one key per intent

The rule that trips people up is that the key belongs to the intent, not the attempt. The client generates it once, when the customer presses Pay, and sends the same value on every attempt of that same action. A key generated inside the retry loop makes every attempt look new and buys nothing.

Stripe's idempotency documentation is the clearest published contract, and worth copying:

• "A client generates an idempotency key … we suggest using V4 UUIDs, or another random string with enough entropy to avoid collisions." Keys can be up to 255 characters. Do not use anything sensitive, such as an email address, as a key.
• "Stripe's idempotency works by saving the resulting status code and body of the first request made for any given idempotency key, regardless of whether it succeeds or fails. Subsequent requests with the same key return the same result, including 500 errors."
• "You can remove keys from the system automatically after they're at least 24 hours old."
• "The idempotency layer compares incoming parameters to those of the original request and errors if they're not the same."

The wire format has a draft standard too. The IETF's Idempotency-Key header draft reached version 07 before expiring, so treat it as a common convention rather than a standard, but its advice is sound and it is what most APIs already do. It recommends a UUID, allows an "idempotency fingerprint … in conjunction with an idempotency key to determine the uniqueness of a request", and sets out the error cases.

Note what is not on that list: a plain 200 with a fresh result. If you reprocess the request and return a new order, the client has no way of knowing a duplicate exists.

Where the key store lives

The common mistake is putting the key in Redis and the order in Postgres. Crash between the two and you own the worst of both: a claimed key with no order behind it, or an order nobody can find. Put the key in the same database as the thing it protects, and claim it in the same transaction as the write.

Postgres gives you exactly the primitive you need. The INSERT documentation promises that "ON CONFLICT DO UPDATE guarantees an atomic INSERT or UPDATE outcome … even under high concurrency", and that RETURNING yields "only rows that were successfully inserted or updated". So an insert that returns no row is the signal that someone else got there first.

create table idempotency_keys (
  key             text primary key,
  request_hash    text        not null,
  state           text        not null check (state in ('running', 'done')),
  response_status int,
  response_body   jsonb,
  claimed_at      timestamptz not null default now()
);

create index idempotency_keys_claimed_at_idx on idempotency_keys (claimed_at);

The handler claims the key first, and only then does the work:

import { createHash } from 'node:crypto';
import type { Request, Response } from 'express';
import { pool } from './db';

type Saved = { status: number; body: unknown };

// Sort keys before hashing: {a,b} and {b,a} are the same request.
const canonical = (value: unknown): unknown => {
  if (Array.isArray(value)) return value.map(canonical);
  if (value && typeof value === 'object') {
    const entries = Object.entries(value as Record<string, unknown>).sort(([a], [b]) => a.localeCompare(b));
    return Object.fromEntries(entries.map(([k, v]) => [k, canonical(v)]));
  }
  return value;
};

const fingerprint = (req: Request): string =>
  createHash('sha256').update(JSON.stringify([req.method, req.path, canonical(req.body)])).digest('hex');

export async function createOrder(req: Request, res: Response): Promise<void> {
  const key = req.header('Idempotency-Key');
  if (!key) {
    res.status(400).json({ error: 'Idempotency-Key header is required' });
    return;
  }

  const hash = fingerprint(req);
  const claim = await pool.query(
    `INSERT INTO idempotency_keys (key, request_hash, state)
     VALUES ($1, $2, 'running')
     ON CONFLICT (key) DO NOTHING
     RETURNING key`,
    [key, hash],
  );

  if (claim.rowCount === 0) {
    const { rows } = await pool.query(
      `SELECT request_hash, state, response_status, response_body
         FROM idempotency_keys WHERE key = $1`,
      [key],
    );
    const seen = rows[0];

    if (seen.request_hash !== hash) {
      res.status(422).json({ error: 'This key was already used for a different request' });
      return;
    }
    if (seen.state === 'running') {
      res.setHeader('Retry-After', '2');
      res.status(409).json({ error: 'The first attempt is still running' });
      return;
    }
    res.status(seen.response_status).json(seen.response_body);
    return;
  }

  const saved = await runOnce(key, req.body);
  res.status(saved.status).json(saved.body);
}

And the work itself commits the order and the saved reply together, so a crash can never leave one without the other:

async function runOnce(key: string, input: { customerId: string; totalCents: number }): Promise<Saved> {
  const client = await pool.connect();
  try {
    await client.query('BEGIN');
    const { rows } = await client.query(
      `INSERT INTO orders (customer_id, total_cents, status)
       VALUES ($1, $2, 'pending')
       RETURNING id`,
      [input.customerId, input.totalCents],
    );

    const saved: Saved = { status: 201, body: { orderId: rows[0].id } };
    await client.query(
      `UPDATE idempotency_keys SET state = 'done', response_status = $2, response_body = $3
        WHERE key = $1`,
      [key, saved.status, saved.body],
    );
    await client.query('COMMIT');
    return saved;
  } catch (error) {
    await client.query('ROLLBACK');
    // Release the claim so the client's next attempt can try properly.
    await pool.query(`DELETE FROM idempotency_keys WHERE key = $1 AND state = 'running'`, [key]);
    throw error;
  } finally {
    client.release();
  }
}

Three things to add before this is production code. A sweeper that deletes rows past your TTL, because the table only grows otherwise. A lease on claimed_at, so a key left running by a process that died can be retaken after a minute instead of answering 409 for ever. And, if the work calls a payment provider, pass the same key onward: stripe-node takes { idempotencyKey } in its request options and sets the Idempotency-Key header from it, so one identity covers the whole chain.

If the work never touches your database — sending an SMS, calling a third party — Redis is a reasonable store. SET with NX and EX is the same atomic claim in one command: SET idem:<key> running EX 86400 NX returns nil when the key is already there. You still have two systems that can disagree, so keep this for work that has no database row of its own.

Queues and webhooks are the same problem

Every queue you use has already made this decision for you. Amazon's SQS documentation is blunt: you "might get that message copy again when you receive messages. Design your applications to be idempotent (they should not be affected adversely when processing the same message more than once)."

Webhooks are the same. Stripe's webhook guidance says endpoints "might occasionally receive the same event more than once" and that you should "guard against duplicated event receipts by logging the event IDs you've processed". It also notes that events are not delivered in order, so you cannot use a timestamp to work out whether you have seen something.

The consumer-side pattern is one insert, sharing a transaction with the work:

import { Worker } from 'bullmq';
import { pool } from './db';
import { applyEvent } from './events';

new Worker('provider-webhooks', async (job) => {
  const { eventId, payload } = job.data;
  const client = await pool.connect();
  try {
    await client.query('BEGIN');
    const first = await client.query(
      `INSERT INTO processed_events (event_id) VALUES ($1)
       ON CONFLICT DO NOTHING
       RETURNING event_id`,
      [eventId],
    );

    if (first.rowCount === 0) {
      await client.query('COMMIT');
      return; // Seen before. Succeed quietly so the queue stops redelivering.
    }

    await applyEvent(client, payload);
    await client.query('COMMIT');
  } catch (error) {
    await client.query('ROLLBACK'); // The dedupe row rolls back with the work.
    throw error;
  } finally {
    client.release();
  }
}, { connection });

The rollback is the part worth staring at. If the dedupe insert committed separately and the work then failed, the retry would find the event "already processed" and skip it for ever. One transaction, or a silent hole in your data.

BullMQ can also dedupe at the door: give a job the event id as its jobId and, per its documentation, "if you add a job with an existing id then that job will just be ignored and not added to the queue at all". Useful, with one caveat from the same page: jobs removed by removeOnComplete are no longer considered duplicates, so this is a short-lived guard, not a record.

Where teams get it wrong

The check-then-insert race deserves its own warning, because it survives every manual test and fails under load. Two concurrent attempts both read "no key", both proceed, both charge. If your deduplication is a get followed by a set, you have a race, not a guarantee. The uniqueness has to be enforced by the database or by a single atomic command.

There is one more failure mode worth naming, because it is invisible until traffic is high: retry amplification. The Builders' Library article on timeouts, retries and backoff describes a five-deep stack of services with three retries at each layer, where "if each layer retries independently, the load on the database will increase 243x, making it unlikely to ever recover". Their rule is to "retry at a single point in the stack". Idempotency makes retries safe; it does not make them free.

Testing it

The only test that matters is the concurrent one. Sequential retries pass on almost any implementation, including the broken check-then-insert version.

import { randomUUID } from 'node:crypto';
import { describe, expect, it } from 'vitest';
import request from 'supertest';
import { app } from '../src/app';
import { pool } from '../src/db';

describe('POST /orders', () => {
  it('creates one order for five concurrent attempts with one key', async () => {
    const key = randomUUID();
    const attempt = () =>
      request(app).post('/orders').set('Idempotency-Key', key).send({ customerId: 'c_1', totalCents: 4000 });

    const replies = await Promise.all(Array.from({ length: 5 }, attempt));

    // One 201, and 200s or 409s for the rest: never two 201s.
    expect(replies.filter((r) => r.status === 201)).toHaveLength(1);
    expect(replies.every((r) => [201, 200, 409].includes(r.status))).toBe(true);

    const { rows } = await pool.query(`SELECT count(*)::int AS n FROM orders WHERE customer_id = 'c_1'`);
    expect(rows[0].n).toBe(1);
  });
});

Run it against a real database, not a mock. The behaviour you are testing lives in the unique constraint, and a mocked store will happily tell you everything is fine.

The short version

A timeout is not a failure. It is an unknown, and the only way to resolve it safely is to retry with something that lets the server recognise the retry. That something is one key per intent, generated by the client, reused by every attempt.

Claim the key with a single atomic insert, do the work and save the reply in the same transaction, and keep the key longer than the client will ever keep trying. Do the same thing on the consumer side with event ids, because your queue already delivers some messages twice. Then test it with concurrent requests against a real database, since that is the only version of the test that can fail.

If you want the arithmetic for your own traffic, the calculator above needs three numbers you already have in your logs: requests a day, timeout rate, and how many of those timeouts turn out to have worked. The rest is deciding whether to keep paying for them. Retry safety also sits underneath availability arithmetic and recovering a backlog, so it is rarely the only thing worth fixing.