How to find and fix flaky Playwright tests

What Playwright calls a flaky test

Playwright sorts every test into one of three groups when retries are enabled: passed (passed on the first run), flaky (failed on the first run, passed when retried) and failed (failed on the first run and on every retry). The same label appears in the terminal (1 flaky), in the HTML report and in a custom reporter, where test.outcome() returns "flaky".

Without retries there is no flaky label at all: a test that fails intermittently simply shows up as failed on some runs and passed on others. That is why the first step is to turn retries on in CI.

Step 1: turn on retries and keep a trace of the retry

Retries are off by default. Enable them in CI only, so a flaky test still fails loudly on your laptop, and record a trace the first time a test is retried:

import { defineConfig } from '@playwright/test';

export default defineConfig({
  retries: process.env.CI ? 2 : 0,
  use: {
    trace: 'on-first-retry',
  },
});

trace: 'on-first-retry' records a trace only for the first retry of a test, so passing runs cost nothing. If you want the trace of the failed attempt itself, use 'retain-on-failure' instead, which records every test and keeps the trace only when it fails.

Step 2: reproduce it on purpose

A flaky test that fails one time in twenty is hard to debug from CI alone. Run it many times in a row without retries:

npx playwright test tests/checkout.spec.ts --repeat-each=20 --retries=0

If it never fails alone, run it together with the rest of its file or the whole suite; a test that fails only in company usually shares state with another test. After a failing run, npx playwright test --last-failed re-runs just the failures.

Step 3: compare the failed attempt with the passing one

Open the trace (npx playwright show-trace trace.zip, or from the HTML report) and step through the actions. Look at the moment the attempts diverge: an element that was not there yet, a request that returned later or with different data, a dialog, a toast covering a button, or a page that was still loading.

The usual causes, and how to fix them

Racing the application

Fixed waits such as page.waitForTimeout(2000) are too short on a slow CI machine and wasted time on a fast one. Use web-first assertions instead: await expect(page.getByRole('button', { name: 'Pay' })).toBeEnabled() retries until the condition is true or the assertion times out. Actions like click() already wait for the element to be visible, stable and enabled.

Shared test data

Tests that run in parallel and use the same user, cart or record will step on each other. Give each test its own data, for example a unique email per test, or include testInfo.workerIndex in the names of records you create.

Order dependence

A test that relies on another test having run first breaks as soon as Playwright schedules them differently, for example on another shard. Make every test set up what it needs (fixtures are the tool for this). If a group of tests really must run in order, mark it with test.describe.configure({ mode: 'serial' }), and know that the whole group is then retried together.

Slow or overloaded CI machines

Too many workers on a small runner makes everything slower, and timeouts start to fire. Before raising a timeout, check the worker count and the machine size. A longer timeout makes the symptom rarer, but the test stays just as slow.

Third-party services, time and animations

Payment sandboxes, analytics and feature-flag services can be slow or down. Mock what you don't own with page.route(). For tests that depend on the current time, page.clock lets you control it instead of waiting for it.

Don't let retries hide the problem

Retries keep the pipeline green, which is exactly why a flaky test can stay flaky for months. Two habits help: read the flaky count in every report, and, once your suite is stable, set failOnFlakyTests: true in the config (or pass --fail-on-flaky-tests) so a flaky test fails the run.

Why one report is not enough

Playwright's report describes a single run. It can tell you that a test was flaky today; it cannot tell you whether the same test was flaky yesterday, how often it happens, whether it started with a particular commit, or whether today's failure on your pull request is new or the same old flake. Those questions need the test's results from many runs, across branches.

TurnSignal keeps that history for you. The Flaky tests page lists every test that was flaky in the last 7 to 90 days, how often, and the exact runs and attempts behind each label. When a flaky test fails on a pull request, it is labelled Known flaky (it was flaky in at least one of its previous 20 results), so nobody spends an hour blaming the wrong change. When someone is working on a fix, you can quarantine the test with a reason, an owner and an end date. Quarantine only changes how TurnSignal counts the failure; Playwright still runs the test and your CI still fails when it fails.

Setup is one reporter line next to your existing reporters; see the docs for GitHub Actions, GitLab and other CI.