A retry worker that loops until success without backoff. Upstream goes degraded; the worker pounds it with 1000+ requests per second; upstream rate-limits the IP; success rate drops to 0% indefinitely.
The vulnerable diff
// workers/sync-retry.ts
async function syncWithRetry() {
// BUG: tight loop, no backoff, no max
while (true) {
try {
await sync();
break;
} catch (err) {
// immediately retry
}
}
}What is wrong
Retry is the right answer for transient failures but the loop needs structure: exponential backoff (delay doubles each attempt), jitter (random component prevents thundering herd), and a max retry count (after which the failure propagates). Without these, a degraded upstream gets a denial-of-service from your own retry loop; once rate-limited, recovery is delayed by the upstream's rate-limit window — sometimes hours.
The attack
Symptom:
t+0:00 — upstream returns 500.
t+0:01 — your worker retries, 500.
t+0:02 — 1000 retries/sec from your worker.
t+0:05 — upstream rate-limits your IP for 1 hour.
t+0:05 — every retry now returns 429.
t+1:05 — rate-limit lifts. By now upstream was healthy for 50 minutes.Self-inflicted outage.
Mesrai's review comment
mesraipilot · Bot · reviewed 30 sec ago
[mesrai] [code-review] [Performance] [Retry-Storm] [medium]
Tight retry loop hammers upstream and triggers rate-limit bans.
Exponential backoff with jitter and a max:
async function withBackoff<T>(
fn: () => Promise<T>,
opts = { max: 5, baseMs: 200 }
): Promise<T> {
for (let i = 0; i < opts.max; i++) {
try { return await fn(); }
catch (err) {
if (i === opts.max - 1) throw err;
const exp = opts.baseMs * 2 ** i;
const jitter = Math.random() * exp * 0.5;
await sleep(exp + jitter);
}
}
throw new Error("unreachable");
}
For high-volume calls, use a circuit breaker (opossum, or
hand-rolled): trip after N consecutive failures, half-open with
single requests to test recovery.
Most retry libraries ship this — p-retry, async-retry, axios-retry.The fix
// workers/sync-retry.ts — fixed
import pRetry from "p-retry";
async function syncWithRetry() {
await pRetry(sync, {
retries: 5,
factor: 2,
minTimeout: 200,
randomize: true,
onFailedAttempt: (err) => log.warn({ err, attempt: err.attemptNumber }, "sync retry"),
});
}p-retry handles backoff + jitter + max. Logs per-attempt. After 5 attempts the underlying error propagates.
Why human review missed it
Tight retry loops are common in worker code. Mesrai catches `while(true) catch` retry patterns and recommends backoff library.
Related rules + further reading
Mesrai rule pack: performance/retry-needs-backoff — flags retry loops without exponential backoff + jitter.
AWS Architecture Blog: Exponential Backoff and Jitter.
p-retry / async-retry / axios-retry libraries.
Takeaway
Retry with backoff + jitter + max. Or use p-retry. Mesrai catches every unbounded retry loop.