Real incident: a refund-retry loop processed the same pending refund 47 times before the alert fired. Total refunded: $40K extra. Recovery: manual reversal at Stripe + bank chargebacks. Mesrai catches the shape at PR time now.

The vulnerable diff

tsbilling/process-refund.ts

// billing/process-refund.ts (the buggy loop)
async function processRefunds() {
  // BUG: every loop iteration creates a new refund — no idempotency
  while (true) {
    const pending = await db.refunds.findMany({ where: { status: "pending" } });
    if (pending.length === 0) break;

    for (const refund of pending) {
      try {
        await stripe.refunds.create({ amount: refund.amountCents, charge: refund.chargeId });
        // BUG: status update is the LAST step but errors above leave status pending
      } catch (err) {
        // retry on next outer loop — but the refund DID succeed at Stripe,
        // we just didn't update our DB
      }
    }
  }
}

What is wrong

The post-mortem identified two cascading bugs. (1) Refund creation succeeded at Stripe but the local status update failed (DB connection blip). (2) The outer retry loop re-read all pending refunds and re-tried — without idempotency, Stripe accepted each retry as a new refund. Each iteration of the outer loop created another refund per record. By the time the on-call engineer paged, 47 retries had run on the worst-affected customer. Stripe refunded them 47 times their original charge.

The attack

Post-mortem timeline:

textexploit

t+0:00 — Worker started. Reads 12 pending refunds.
t+0:01 — Refund 1 succeeds at Stripe. DB update fails (connection blip).
t+0:01 — Outer loop sees same refund still "pending" — retries.
t+0:30 — Customer has received refund 12 times.
t+1:00 — Customer has received refund 47 times.
t+1:00 — Stripe Radar alerts "duplicate refund frequency".
t+1:05 — Pager. Worker paused.
t+24:00 — Stripe processes reverse-refunds; some go through, some bounce
          back as bank chargebacks. Cost: ~$40K + ~$2K in fees.

Recovery involved both engineering and finance teams.

Mesrai's review comment

textreview-comment.txt

mesraipilot · Bot · reviewed 1 min ago

[mesrai] [code-review] [Logic] [Idempotency] [critical]

Refund retry loop without idempotency keys. Stripe refunds.create
without an idempotency key creates a new refund on every call —
retries duplicate the refund.

Fix:

  for (const refund of pending) {
    const key = `refund:${refund.id}:${refund.attempt}`;
    try {
      await stripe.refunds.create(
        { amount: refund.amountCents, charge: refund.chargeId },
        { idempotencyKey: key }
      );
      await db.refunds.update({
        where: { id: refund.id },
        data: { status: "succeeded", processedAt: new Date() },
      });
    } catch (err) {
      // Increment attempt counter; next retry uses different key only
      // if you want a fresh attempt, otherwise SAME key for safe retry.
      await db.refunds.update({
        where: { id: refund.id },
        data: { attempt: { increment: 1 }, lastError: err.message },
      });
    }
  }

For repeated retries: Stripe idempotency keys are valid for 24 hours;
same key = same result returned, no new refund created.

The fix

tsbilling/process-refund.ts (fixed)

// billing/process-refund.ts — fixed
async function processRefunds() {
  const pending = await db.refunds.findMany({ where: { status: "pending" } });
  for (const refund of pending) {
    const idempotencyKey = `refund:${refund.id}`;  // one stable key per refund record
    try {
      await stripe.refunds.create(
        { amount: refund.amountCents, charge: refund.chargeId },
        { idempotencyKey }
      );
      await db.refunds.update({
        where: { id: refund.id },
        data: { status: "succeeded", processedAt: new Date() },
      });
    } catch (err) {
      log.error({ err, refundId: refund.id }, "refund failed");
      // Stay pending; next worker run will retry with the same key.
    }
  }
}

Stable idempotency key per refund record. Stripe deduplicates by key. Retries safe. Move DB status update to happen explicitly after Stripe ack, and accept that the refund stays pending on errors until the next worker pass — which will retry safely.

Why human review missed it

Refund + idempotency bugs are catastrophic. Mesrai catches every Stripe / Razorpay / payment-provider create call without an explicit idempotency key.

Related rules + further reading

Mesrai rule pack: logic/payment-provider-idempotency — flags payment provider create calls without idempotency.

Stripe Idempotent Requests doc — the canonical reference.

Cluster 3 post 27 + Cluster 7 post 64 both cover the same shape from different angles.

Takeaway

Refund loops without idempotency cost real money. Stable key per record. Mesrai catches the absence.

The Payment Refund Loop That Cost $40K — Caught Pre-Merge by Mesrai