Track Playwright test history across CI runs

What Playwright gives you for one run

Playwright's reporters are built around a single run. The html reporter is a browsable report of that run. The json and junit reporters write the same run as a file for other tools. The blob reporter exists to merge the shards of one run into one report with npx playwright merge-reports. None of them reads earlier runs.

Inside one run you already get useful labels. With retries on, a test that failed and then passed is reported as flaky, and one that failed every attempt as failed. What you cannot see is the pattern: whether the same test was flaky yesterday, how often, or whether today's failure on your branch is also happening on main.

Step 1: save a JSON report from every run

Add the JSON reporter next to the reporters you already use, writing to a file. In Playwright 1.63 the option is outputFile; the PLAYWRIGHT_JSON_OUTPUT_FILE environment variable does the same from the command line.

// playwright.config.ts
import { defineConfig } from '@playwright/test';

export default defineConfig({
  retries: process.env.CI ? 2 : 0,
  reporter: [
    ['list'],
    ['html', { open: 'never' }],
    ['json', { outputFile: 'results/results.json' }],
  ],
});

Retries matter here: without them Playwright never labels a test flaky, and your history can only say passed or failed.

Step 2: keep the files longer than CI does

CI artifacts expire. Playwright's own GitHub Actions template keeps the report for 30 days with retention-days: 30, and many teams shorten that to save storage. For history you want months, so copy each run's results.json somewhere durable after the tests, named by date and commit: an object storage bucket, a separate branch, or a shared drive. Keep the branch name with it, because the most useful comparison later is against your default branch.

# after the test step, even when tests fail
mkdir -p history
cp results/results.json "history/$(date -u +%Y%m%dT%H%M%S)-$GITHUB_SHA.json"

Step 3: read the history with a small script

The JSON report nests suites (files, then describe blocks), specs (tests) and one entry per project, each with a status of expected, unexpected, flaky or skipped. This script walks every saved report, keys each test by project, file and title path, and prints the tests that were flaky or failed in any run:

// history.mjs: node history.mjs history/*.json
import { readFileSync } from 'node:fs';

const history = new Map(); // test -> [{ run, status }]

for (const file of process.argv.slice(2)) {
  const report = JSON.parse(readFileSync(file, 'utf8'));
  const run = report.stats.startTime;
  const walk = (suite, titles) => {
    for (const spec of suite.specs) {
      for (const test of spec.tests) {
        const key = [test.projectName, spec.file, ...titles, spec.title].filter(Boolean).join(' › ');
        if (!history.has(key)) history.set(key, []);
        history.get(key).push({ run, status: test.status });
      }
    }
    for (const child of suite.suites ?? []) walk(child, [...titles, child.title]);
  };
  for (const suite of report.suites) walk(suite, []);
}

for (const [test, runs] of history) {
  const flaky = runs.filter((r) => r.status === 'flaky').length;
  const failed = runs.filter((r) => r.status === 'unexpected').length;
  if (flaky || failed) console.log(`${flaky} flaky, ${failed} failed in ${runs.length} runs: ${test}`);
}

We ran it against two JSON reports from Playwright 1.63 and it printed lines like 2 flaky, 0 failed in 2 runs: checkout.spec.ts › checkout › shows order confirmation. Sort that output by the flaky count and you have a list of the tests to fix first, with evidence.

Where the do-it-yourself history breaks

• Renamed or moved tests start a new history, because the key is the title path. Keep renames rare, or map old names by hand.
• Sharded runs write one JSON report per shard. Merge them first (blob reports and merge-reports --reporter json), or the script counts each shard as a separate run.
• Crashed runners leave no file. If a CI machine runs out of memory or is killed, its report is never written, and those tests simply vanish from the history. The guide to missing tests after a crashed shard shows why this happens.
• "Is it failing on main too?" needs the branch of every run and a rule for comparing: the latest result on the default branch, per test. That is more code, and it is the question people ask most on a red pull request.
• Someone has to run it. A script answers questions when someone remembers to run it, not on the pull request where the failure shows up.

For a small suite and a few runs a day, the script is often enough. As the suite and the number of pull requests grow, these gaps are where teams lose time.

Hosted history without changing your reporters

TurnSignal keeps this history for you. It runs as one more reporter next to the HTML report. Every run opens with a verdict, every failure on a pull request is labelled new, also failing on main or known flaky (flaky in at least one of its previous 20 results), shards of one pipeline merge into one run, and tests from a runner that crashed are listed as Missing instead of disappearing. Results are written to disk in CI before upload, and the reporter never changes your exit code. On GitHub the verdict is posted as a pull request comment. Setup is in the docs; it is free during the public beta.