~/speedsharma.com

writing/architecture

Idempotency in Distributed Systems — Making Any Operation Safe to Retry

Learn idempotency key design, idempotency stores with TTL, request fingerprinting, and CQRS deduplication patterns for safe retries.

S Sanjeev Sharma
15 March 2026 · 2 min read

Introduction

Network failures are inevitable. A client retries a request, which succeeds, but the response is lost. Without idempotency, that retry creates a duplicate charge, duplicate order, or duplicate message. Idempotency is your shield: the same request executed multiple times produces the same result as executing once. We'll explore idempotency key design, storage strategies, and how to verify duplicates at scale.

Idempotency Key Design

An idempotency key uniquely identifies a request intent. It can be client-generated or deterministically derived.

Client-generated with UUID:

interface CreateChargeRequest {
  amount: number;
  currency: string;
  customerId: string;
  idempotencyKey: string; // Client provides UUID
}

class PaymentAPI {
  async createCharge(req: CreateChargeRequest): Promise<ChargeResponse> {
    const normalized = {
      method: 'POST',
      path: '/charges',
      body: req,
      idempotencyKey: req.idempotencyKey,
    };

    // Check if we've seen this idempotency key
    const cached = await this.idempotencyStore.get(req.idempotencyKey);
    if (cached) {
      return cached.response; // Return previous response
    }

    const charge = await this.stripe.charges.create({
      amount: req.amount,
      currency: req.currency,
      customer: req.customerId,
    });

    const response: ChargeResponse = {
      chargeId: charge.id,
      amount: charge.amount,
      status: charge.status,
    };

    // Store result with TTL (24 hours)
    await this.idempotencyStore.set(req.idempotencyKey, { response, timestamp: Date.now() }, 86400);

    return response;
  }
}

interface ChargeResponse {
  chargeId: string;
  amount: number;
  status: 'succeeded' | 'pending' | 'failed';
}

Deterministic hashing from request content:


class DeterministicIdempotency {
  generateKey(method: string, path: string, body: any): string {
    const normalized = JSON.stringify({
      method,
      path,
      body: this.normalizeBody(body),
    });

    return crypto.createHash('sha256').update(normalized).digest('hex');
  }

  private normalizeBody(body: any): any {
    // Ensure consistent ordering and structure
    if (Array.isArray(body)) {
      return body.map(item => this.normalizeBody(item));
    }
    if (typeof body === 'object' && body !== null) {
      return Object.keys(body)
        .sort()
        .reduce((acc, key) => {
          acc[key] = this.normalizeBody(body[key]);
          return acc;
        }, {} as Record<string, any>);
    }
    return body;
  }

  async createTransfer(req: any): Promise<any> {
    const idempotencyKey = this.generateKey('POST', '/transfers', req);

    const cached = await this.idempotencyStore.get(idempotencyKey);
    if (cached) {
      return cached;
    }

    const result = await this.processTransfer(req);
    await this.idempotencyStore.set(idempotencyKey, result, 86400);
    return result;
  }

  private async processTransfer(req: any): Promise<any> {
    // Implementation
    return {};
  }
}

Idempotency Store with TTL

Redis is ideal for idempotency stores: fast, supports TTL, and atomic operations.

class RedisIdempotencyStore {
  constructor(private redis: Redis) {}

  async get(key: string): Promise<any> {
    const value = await this.redis.get(`idempotent:${key}`);
    if (!value) return null;
    try {
      return JSON.parse(value);
    } catch {
      return null;
    }
  }

  async set(key: string, value: any, ttlSeconds: number): Promise<void> {
    const serialized = JSON.stringify({
      response: value,
      storedAt: Date.now(),
    });
    await this.redis.setex(`idempotent:${key}`, ttlSeconds, serialized);
  }

  async exists(key: string): Promise<boolean> {
    return (await this.redis.exists(`idempotent:${key}`)) === 1;
  }

  async delete(key: string): Promise<void> {
    await this.redis.del(`idempotent:${key}`);
  }

  // For cleanup: scan and remove expired keys
  async cleanup(): Promise<number> {
    const cursor = '0';
    let cleaned = 0;
    const scan = async (c: string): Promise<string> => {
      const [newCursor, keys] = await this.redis.scan(c, 'MATCH', 'idempotent:*', 'COUNT', 100);
      for (const key of keys) {
        const ttl = await this.redis.ttl(key);
        if (ttl === -1) {
          // No TTL set, delete it
          await this.redis.del(key);
          cleaned++;
        }
      }
      return newCursor;
    };

    let currentCursor = cursor;
    do {
      currentCursor = await scan(currentCursor);
    } while (currentCursor !== '0');

    return cleaned;
  }
}

Database-Level Upsert Patterns

For operations that must update state (not just read), use database upserts with unique constraints.

class OrderService {
  async createOrder(req: CreateOrderRequest): Promise<Order> {
    const idempotencyKey = req.idempotencyKey;

    // Upsert with ON CONFLICT
    const result = await this.db.query(
      `INSERT INTO orders (idempotency_key, customer_id, total, status, created_at)
       VALUES ($1, $2, $3, 'pending', NOW())
       ON CONFLICT (idempotency_key)
       DO UPDATE SET updated_at = NOW()
       RETURNING id, customer_id, total, status, created_at`,
      [idempotencyKey, req.customerId, req.totalAmount]
    );

    const order = result.rows[0];

    // If this is a retry, the INSERT will be skipped and we return existing order
    // Ensure the order is not already processed
    if (order.status !== 'pending') {
      return order;
    }

    // Proceed with order processing
    await this.db.query(
      `UPDATE orders SET status = 'processing' WHERE id = $1`,
      [order.id]
    );

    return order;
  }

  async updateOrderStatus(orderId: string, idempotencyKey: string, status: string): Promise<void> {
    const result = await this.db.query(
      `INSERT INTO order_status_changes (order_id, idempotency_key, new_status, changed_at)
       VALUES ($1, $2, $3, NOW())
       ON CONFLICT (order_id, idempotency_key)
       DO NOTHING
       RETURNING id`,
      [orderId, idempotencyKey, status]
    );

    // Only update if this is the first time we've seen this idempotency key
    if (result.rows.length > 0) {
      await this.db.query(`UPDATE orders SET status = $1 WHERE id = $2`, [status, orderId]);
    }
  }
}

Idempotent Webhook Processing

Webhooks are retried on failure. Process them idempotently.

interface WebhookEvent {
  eventId: string; // Unique across all events
  eventType: string;
  timestamp: number;
  data: any;
}

class WebhookProcessor {
  async handleWebhook(event: WebhookEvent): Promise<void> {
    // Use eventId as idempotency key
    const cached = await this.idempotencyStore.get(`webhook:${event.eventId}`);
    if (cached) {
      return; // Already processed
    }

    try {
      await this.processEvent(event);
      await this.idempotencyStore.set(`webhook:${event.eventId}`, { processed: true }, 86400);
    } catch (error) {
      // Don't cache errors; allow retry
      throw error;
    }
  }

  private async processEvent(event: WebhookEvent): Promise<void> {
    switch (event.eventType) {
      case 'payment.completed':
        await this.handlePaymentCompleted(event.data);
        break;
      case 'payment.failed':
        await this.handlePaymentFailed(event.data);
        break;
      default:
        throw new Error(`Unknown event type: ${event.eventType}`);
    }
  }

  private async handlePaymentCompleted(data: any): Promise<void> {
    // Safe to call multiple times
    await this.db.query(
      `UPDATE orders SET status = 'paid', paid_at = NOW()
       WHERE payment_id = $1 AND status != 'paid'`,
      [data.paymentId]
    );
  }

  private async handlePaymentFailed(data: any): Promise<void> {
    await this.db.query(
      `UPDATE orders SET status = 'payment_failed' WHERE payment_id = $1`,
      [data.paymentId]
    );
  }
}

CQRS Command Deduplication

In CQRS, commands are deduplicated at the command handler level.

interface Command {
  commandId: string; // Unique command identifier
  type: string;
  payload: any;
  timestamp: number;
}

class CommandHandler {
  private commandStore = new Map<string, any>(); // or Redis

  async execute<T>(command: Command): Promise<T> {
    // Check if command was already executed
    const result = await this.getCommandResult(command.commandId);
    if (result !== undefined) {
      return result as T;
    }

    // Execute the command
    const commandResult = await this.executeCommand(command);

    // Store result
    await this.storeCommandResult(command.commandId, commandResult);

    return commandResult as T;
  }

  private async executeCommand(command: Command): Promise<any> {
    switch (command.type) {
      case 'CreateAccount':
        return await this.createAccount(command.payload);
      case 'DepositFunds':
        return await this.depositFunds(command.payload);
      case 'WithdrawFunds':
        return await this.withdrawFunds(command.payload);
      default:
        throw new Error(`Unknown command: ${command.type}`);
    }
  }

  private async createAccount(payload: any): Promise<string> {
    const accountId = await this.db.query(
      `INSERT INTO accounts (owner_id, balance, created_at)
       VALUES ($1, $2, NOW())
       RETURNING id`,
      [payload.ownerId, payload.initialBalance]
    );
    return accountId.rows[0].id;
  }

  private async depositFunds(payload: any): Promise<void> {
    await this.db.query(
      `UPDATE accounts SET balance = balance + $1 WHERE id = $2`,
      [payload.amount, payload.accountId]
    );
  }

  private async withdrawFunds(payload: any): Promise<void> {
    const result = await this.db.query(
      `UPDATE accounts SET balance = balance - $1
       WHERE id = $2 AND balance >= $1
       RETURNING balance`,
      [payload.amount, payload.accountId]
    );
    if (result.rows.length === 0) {
      throw new Error('Insufficient funds');
    }
  }

  private async getCommandResult(commandId: string): Promise<any> {
    const result = await this.db.query(
      `SELECT result FROM command_results WHERE command_id = $1`,
      [commandId]
    );
    return result.rows[0]?.result;
  }

  private async storeCommandResult(commandId: string, result: any): Promise<void> {
    await this.db.query(
      `INSERT INTO command_results (command_id, result, executed_at)
       VALUES ($1, $2, NOW())
       ON CONFLICT (command_id) DO NOTHING`,
      [commandId, JSON.stringify(result)]
    );
  }
}

Testing Idempotency with Concurrent Requests

describe('Idempotency', () => {
  it('should deduplicate concurrent identical requests', async () => {
    const service = new PaymentService(store, gateway);
    const idempotencyKey = uuid();

    const promises = Array(5)
      .fill(null)
      .map(() =>
        service.createCharge({
          amount: 1000,
          currency: 'USD',
          customerId: 'cust-123',
          idempotencyKey,
        })
      );

    const results = await Promise.all(promises);

    // All should return same charge ID
    const chargeIds = new Set(results.map(r => r.chargeId));
    expect(chargeIds.size).toBe(1);

    // Only one charge should exist in system
    const charges = await gateway.listCharges({ customerId: 'cust-123' });
    expect(charges.filter(c => c.amount === 1000)).toHaveLength(1);
  });

  it('should return 409 Conflict when retry with same key completes later', async () => {
    const service = new PaymentService(store, gateway);
    const idempotencyKey = uuid();

    const first = service.createCharge({
      amount: 2000,
      currency: 'USD',
      customerId: 'cust-456',
      idempotencyKey,
    });

    const second = service.createCharge({
      amount: 2000,
      currency: 'USD',
      customerId: 'cust-456',
      idempotencyKey,
    });

    const [result1, result2] = await Promise.all([first, second]);
    expect(result1.chargeId).toBe(result2.chargeId);
  });
});

HTTP Status Codes for Duplicates

  • 200 OK: Retry of successful request; return cached response
  • 202 Accepted: Request still processing; advise client to retry
  • 409 Conflict: Different request with same idempotency key; reject with details
class HTTPHandler {
  async post(req: Request): Promise<Response> {
    const idempotencyKey = req.headers['idempotency-key'];
    if (!idempotencyKey) {
      return Response.json({ error: 'Missing idempotency-key' }, { status: 400 });
    }

    const inProgress = await this.store.isInProgress(idempotencyKey);
    if (inProgress) {
      return Response.json({ error: 'Request in progress' }, { status: 202 });
    }

    const existing = await this.store.get(idempotencyKey);
    if (existing) {
      if (existing.requestHash === this.hashRequest(req)) {
        return Response.json(existing.response, { status: 200 });
      } else {
        return Response.json(
          { error: 'Different request with same idempotency key' },
          { status: 409 }
        );
      }
    }

    const response = await this.processRequest(req);
    await this.store.set(idempotencyKey, { response, requestHash: this.hashRequest(req) });
    return Response.json(response, { status: 201 });
  }

  private hashRequest(req: Request): string {
    return crypto
      .createHash('sha256')
      .update(JSON.stringify({ body: req.body }))
      .digest('hex');
  }
}

Checklist

  • Require idempotency keys in all API requests that mutate state
  • Store idempotency results with TTL (24+ hours)
  • Use unique database constraints for insert operations
  • Verify all operations are truly idempotent (no side effects on retry)
  • Test with concurrent retries to catch race conditions
  • Return appropriate HTTP status codes (200, 202, 409)
  • Monitor idempotency cache hit rate
  • Document which endpoints require idempotency keys

Conclusion

Idempotency transforms retries from a liability into a safety net. Design your keys thoughtfully, store results durably, and use database constraints to prevent accidental duplicates. Test concurrency aggressively—the production network will find your assumptions wrong.

S

Sanjeev Sharma

Product Engineer at Acefone, building real-time communications at carrier scale. Author of Logixia and Where Code Runs. I take on a small number of consulting engagements each year.

work with me → github x / twitter

keep reading

architecture · 1 min

Overengineering with Microservices Too Early — When Complexity Kills Speed

You split your MVP into 12 microservices before you had 100 users. Now a simple feature requires coordinating 4 teams, 6 deployments, and debugging across 8 services. The architecture that was supposed to scale you faster is the reason you ship slower than your competitors.

Mar 2026

ai engineering · 2 min

LLM Observability in Production — Tracing Every Token From Request to Response

Master end-to-end LLM observability with OpenTelemetry spans, trace correlation, adaptive sampling, and anomaly detection to catch production issues before users do.

Mar 2026

observability · 1 min

Logging Everything and Nothing Useful — The Noise Problem

Your logs are full. Gigabytes per hour. Health check pings, SQL query text, Redis GET/SET for every cached value. When a real error occurs, it's buried under 50,000 noise lines. You log everything and still can't find what you need in a production incident.

Mar 2026