Flaky tests: what they are, why they happen, and how to fix them

What is a flaky test?

A test is flaky when its result isn't determined by the code under test alone. Run it twice on the same commit and it can pass once and fail once. A test that fails every time is broken, not flaky — and much easier to fix.

Flakiness can live in the test (a fixed sleep, shared fixtures), in the environment (a slow CI runner, a busy port) or in the product itself (a real race condition that shows up first as an intermittent test failure).

Why flaky tests are expensive

Each flaky failure costs a re-run and a developer's attention. The bigger cost is trust: once a team learns that red builds are often noise, real failures get re-run instead of investigated, and bugs slip through.

The problem grows with test suites. Google reported in 2016 that almost 16% of its tests showed some level of flakiness, and that 84% of the transitions it observed from passing to failing involved a flaky test — at that scale, flakiness becomes an engineering cost of its own.

What causes flaky tests?

An empirical study of flaky tests in open-source projects (Luo et al., FSE 2014) analysed 201 commits that fixed flaky tests across 51 Apache projects. Asynchronous waiting was the most common root cause — about 45% — followed by concurrency and test-order dependency. In practice, look at these causes first:

Example: replacing a sleep with a real wait

Fixed sleeps are the classic cause of flakiness: too short on a slow CI runner, wasted time on a fast one.

test("export completes", async () => {
  startExport(reportId);
  await sleep(500);
  expect(await getStatus(reportId)).toBe("done");
});

// Deterministic: wait for the condition, with a timeout
test("export completes", async () => {
  startExport(reportId);
  await waitFor(async () => expect(await getStatus(reportId)).toBe("done"), {
    timeout: 10_000,
  });
});

How to detect flaky tests

• Rerun failures and compare. A test that fails then passes on the same commit is flaky by definition. Record every such case.
• Track failure history per test. Test analytics in CI — for example CircleCI test insights, Datadog CI Visibility, Buildkite Test Engine or Trunk — flag tests with inconsistent results over time.
• Run tests in random order. Exposes hidden order dependencies: jest --randomize, pytest-randomly, go test -shuffle=on, RSpec --order random.
• Stress-test suspects. Run one suspicious test many times in parallel and under load: go test -count=100 -race, playwright test --repeat-each=50.

How to fix flaky tests in common frameworks

• Playwright. Rely on auto-waiting and web-first assertions such as await expect(locator).toHaveText(…) instead of manual waits; select elements by role or test id; record a trace on the first retry (trace: 'on-first-retry') to see exactly what happened.
• Cypress. Avoid cy.wait(milliseconds); intercept the request with cy.intercept and wait on its alias; select elements with data-cy attributes; reset state in beforeEach.
• Jest. Await every promise before asserting; use jest.useFakeTimers() for timers and dates; isolate module state with jest.resetModules(); find order dependencies with --randomize.
• pytest. Use function-scoped fixtures for mutable state; control time with freezegun; seed random data; find order dependencies with pytest-randomly.
• Go. Run with -race to find data races; avoid package-level state in tests; use t.Cleanup; check order dependence with -shuffle=on.

Quarantine, fix or delete?

When a test is confirmed flaky, quarantine it so it stops blocking merges — but keep running it and recording results, and open a ticket with its failure history and an owner. Set a deadline: a quarantined test that nobody fixes within a sprint or two should either be rewritten or deleted deliberately, not left in limbo.

Avoid blanket automatic retries. They make the build green, but they also hide real race conditions in the product — sometimes the test is flaky because the feature is. Retries are a temporary measure for a quarantined test, not a fix.

Flaky tests in end-to-end suites

End-to-end tests are the most flake-prone because they cross the most boundaries: browser, network, backend, database and third-party services. Keep them few and focused on critical user journeys, give each its own test data, stub third-party services you don't control, and push detailed logic checks down into unit and integration tests, which are faster and more deterministic.

Preventing flaky tests in code review

Most flakiness is visible in the test code when it's written: fixed sleeps, missing awaits, real timestamps, shared fixtures, network calls and assertions on unordered results. Reviewing new tests for these patterns is far cheaper than debugging an intermittent CI failure later — and the same review catches the race conditions in product code that often appear first as 'flaky' tests.

Flaky test review checklist

• No fixed sleeps — tests wait on a condition with a timeout.
• Every async operation is awaited before assertions.
• Each test creates and cleans up its own data.
• No dependence on test order, global state or shared caches.
• Time is controlled with a fake clock; timezone is explicit.
• Random data is seeded.
• No real network calls to third-party services.
• UI tests select elements by role, label or test id.
• Assertions don't depend on the order of unordered collections.