Real incident: a network glitch caused the payment provider to retry a `payment.captured` webhook 4 times within 30 seconds. The handler created 4 orders for the same payment. Customer service handled the refund cycle. Mesrai catches the shape on every webhook PR now.

The vulnerable diff

tswebhooks/handle-payment.ts

// webhooks/handle-payment.ts (the buggy handler)
paymentRouter.post("/webhook", verifySignature, async (req, res) => {
  const event = JSON.parse(req.body);
  // BUG: no event-id idempotency
  if (event.type === "payment.captured") {
    await createOrder({
      userId: event.payload.userId,
      amount: event.payload.amount,
      chargeId: event.payload.id,
    });
  }
  res.status(200).end();
});

What is wrong

Same shape as cluster 3 post 30, real-incident version. Webhook providers (Stripe, Razorpay, GitHub, etc.) retry on transient failures (network blip, slow response, 5xx response). Without dedupe by event id, the handler processes each delivery as a unique event — creating duplicate orders, duplicate emails, duplicate side effects. Recovery involves manually reversing the duplicates and apologizing.

The attack

Post-mortem:

textexploit

t+0:00 — Network glitch in provider's region.
t+0:01 — Provider sends payment.captured event. Handler 200s but response slow.
t+0:03 — Provider doesn't see 200 in time. Retries.
t+0:05 — Three more retries within 30s.
t+0:35 — Customer receives 4 orders, 4 confirmation emails.
t+0:40 — Customer support ticket.
t+1:00 — Pager. Discover 4 orders for 1 payment.
t+1:30 — Reverse 3 of 4. Customer apology email.

Same shape that caused incidents at every webhook-driven business.

Mesrai catches the shape now.

Mesrai's review comment

textreview-comment.txt

mesraipilot · Bot · reviewed 1 min ago

[mesrai] [code-review] [Logic] [Webhook-Idempotency] [critical]

Webhook handler does not dedupe by event.id. Provider retries on
transient failures — duplicate side effects per retry.

Fix with a persistent event-id table:

  paymentRouter.post("/webhook", verifySignature, async (req, res) => {
    const event = JSON.parse(req.body);

    try {
      await db.webhookEvents.create({
        data: { id: event.id, type: event.type, receivedAt: new Date() },
      });
    } catch (e: any) {
      if (e.code === "P2002") {
        // Already processed — return 200 to acknowledge.
        return res.status(200).end();
      }
      throw e;
    }

    if (event.type === "payment.captured") {
      await createOrder({ ... });
    }
    res.status(200).end();
  });

The unique-constraint violation IS the dedupe check.

The fix

tswebhooks/handle-payment.ts (fixed)

// webhooks/handle-payment.ts — fixed
paymentRouter.post("/webhook", verifySignature, async (req, res) => {
  const event = JSON.parse(req.body);
  try {
    await db.webhookEvents.create({
      data: { id: event.id, type: event.type, receivedAt: new Date() },
    });
  } catch (e: any) {
    if (e.code === "P2002") return res.status(200).end();
    throw e;
  }
  if (event.type === "payment.captured") {
    await createOrder({
      userId: event.payload.userId, amount: event.payload.amount, chargeId: event.payload.id,
    });
  }
  res.status(200).end();
});

Event-id table as idempotency. Unique-constraint violation on duplicate = bail with 200. Side effects guaranteed once per event.

Why human review missed it

Webhook handlers without idempotency are responsible for most duplicate-charge / duplicate-order incidents. Mesrai catches every webhook handler missing the dedupe pattern.

Related rules + further reading

Mesrai rule pack: logic/webhook-event-id-dedupe — flags webhook handlers without persistent event-id idempotency.

Cluster 3 post 30 — same shape, original treatment.

Provider docs (Stripe, Razorpay) all document the at-least-once delivery semantics.

Takeaway

Webhook retries are normal. Idempotency by event-id is required. Mesrai catches every missing dedupe.

A Race Condition in Webhook Replay That Created Duplicate Orders