# Track Playwright test history across CI runs

> Playwright reports describe one run. How to keep test history across CI runs: JSON reports, a small script to find flaky and failing tests, and its limits.

Published 2026-10-12, updated 2026-10-12. Canonical: https://turnsignal.ai/blog/playwright-test-history-across-ci-runs

## In short

- Every Playwright reporter describes a single run. The HTML report, JSON and JUnit files have no memory of yesterday's run or of your main branch.
- To get history, save a machine-readable report from every CI run (the JSON reporter is easiest) and keep the files somewhere longer-lived than CI artifacts.
- A 30-line Node script can then list, per test, how many runs were flaky or failed. That answers "is this test flaky?" with evidence instead of memory.
- The script breaks down on renamed tests, sharded runs, crashed runners and comparisons with main. That is the point where a hosted history pays off.

## What Playwright gives you for one run

Playwright's reporters are built around a single run. The **html** reporter is a browsable report of that run. The **json** and **junit** reporters write the same run as a file for other tools. The **blob** reporter exists to merge the shards of one run into one report with `npx playwright merge-reports`. None of them reads earlier runs.

Inside one run you already get useful labels. With retries on, a test that failed and then passed is reported as **flaky**, and one that failed every attempt as **failed**. What you cannot see is the pattern: whether the same test was flaky yesterday, how often, or whether today's failure on your branch is also happening on main.

## Step 1: save a JSON report from every run

Add the JSON reporter next to the reporters you already use, writing to a file. In Playwright 1.63 the option is `outputFile`; the `PLAYWRIGHT_JSON_OUTPUT_FILE` environment variable does the same from the command line.

```ts
// playwright.config.ts
import { defineConfig } from '@playwright/test';

export default defineConfig({
  retries: process.env.CI ? 2 : 0,
  reporter: [
    ['list'],
    ['html', { open: 'never' }],
    ['json', { outputFile: 'results/results.json' }],
  ],
});
```

Retries matter here: without them Playwright never labels a test flaky, and your history can only say passed or failed.

## Step 2: keep the files longer than CI does

CI artifacts expire. Playwright's own GitHub Actions template keeps the report for 30 days with `retention-days: 30`, and many teams shorten that to save storage. For history you want months, so copy each run's `results.json` somewhere durable after the tests, named by date and commit: an object storage bucket, a separate branch, or a shared drive. Keep the branch name with it, because the most useful comparison later is against your default branch.

```bash
# after the test step, even when tests fail
mkdir -p history
cp results/results.json "history/$(date -u +%Y%m%dT%H%M%S)-$GITHUB_SHA.json"
```

## Step 3: read the history with a small script

The JSON report nests suites (files, then `describe` blocks), specs (tests) and one entry per project, each with a `status` of `expected`, `unexpected`, `flaky` or `skipped`. This script walks every saved report, keys each test by project, file and title path, and prints the tests that were flaky or failed in any run:

```js
// history.mjs: node history.mjs history/*.json
import { readFileSync } from 'node:fs';

const history = new Map(); // test -> [{ run, status }]

for (const file of process.argv.slice(2)) {
  const report = JSON.parse(readFileSync(file, 'utf8'));
  const run = report.stats.startTime;
  const walk = (suite, titles) => {
    for (const spec of suite.specs) {
      for (const test of spec.tests) {
        const key = [test.projectName, spec.file, ...titles, spec.title].filter(Boolean).join(' › ');
        if (!history.has(key)) history.set(key, []);
        history.get(key).push({ run, status: test.status });
      }
    }
    for (const child of suite.suites ?? []) walk(child, [...titles, child.title]);
  };
  for (const suite of report.suites) walk(suite, []);
}

for (const [test, runs] of history) {
  const flaky = runs.filter((r) => r.status === 'flaky').length;
  const failed = runs.filter((r) => r.status === 'unexpected').length;
  if (flaky || failed) console.log(`${flaky} flaky, ${failed} failed in ${runs.length} runs: ${test}`);
}
```

We ran it against two JSON reports from Playwright 1.63 and it printed lines like `2 flaky, 0 failed in 2 runs: checkout.spec.ts › checkout › shows order confirmation`. Sort that output by the flaky count and you have a list of the tests to fix first, with evidence.

## Where the do-it-yourself history breaks

- **Renamed or moved tests** start a new history, because the key is the title path. Keep renames rare, or map old names by hand.
- **Sharded runs** write one JSON report per shard. Merge them first (blob reports and `merge-reports --reporter json`), or the script counts each shard as a separate run.
- **Crashed runners leave no file.** If a CI machine runs out of memory or is killed, its report is never written, and those tests simply vanish from the history. The [guide to missing tests after a crashed shard](https://turnsignal.ai/blog/playwright-missing-tests-crashed-shard) shows why this happens.
- **"Is it failing on main too?"** needs the branch of every run and a rule for comparing: the latest result on the default branch, per test. That is more code, and it is the question people ask most on a red pull request.
- **Someone has to run it.** A script answers questions when someone remembers to run it, not on the pull request where the failure shows up.

For a small suite and a few runs a day, the script is often enough. As the suite and the number of pull requests grow, these gaps are where teams lose time.

## Hosted history without changing your reporters

[TurnSignal](https://turnsignal.ai/product) keeps this history for you. It runs as one more reporter next to the HTML report. Every run opens with a verdict, every failure on a pull request is labelled **new**, **also failing on main** or **known flaky** (flaky in at least one of its previous 20 results), shards of one pipeline merge into one run, and tests from a runner that crashed are listed as **Missing** instead of disappearing. Results are written to disk in CI before upload, and the reporter never changes your exit code. On GitHub the verdict is posted as a pull request comment. Setup is in the [docs](https://turnsignal.ai/docs); it is free during the public beta.

## FAQ

**Does Playwright keep test history between runs?**

No. Each reporter describes one run. To see history, save a report from every run (for example with the JSON reporter) and compare them, or send results to a service that keeps them.

**Which Playwright reporter is best for building history?**

The JSON reporter: it includes every test with its project, file, title path and a status of expected, unexpected, flaky or skipped. JUnit XML works too but carries less detail about retries.

**How do I find the flakiest Playwright tests?**

Turn on retries in CI, save a JSON report from every run, and count per test how many runs reported it as flaky. Sort by that count.

**Why do some tests disappear from my history?**

Usually a renamed test (new title, new key) or a CI runner that crashed before writing its report. The second case hides tests that never ran.

## Sources

- [Playwright docs: Reporters (JSON, JUnit, HTML, blob)](https://playwright.dev/docs/test-reporters)
- [Playwright docs: Retries (flaky vs failed)](https://playwright.dev/docs/test-retries)
- [Playwright docs: Sharding and merging reports](https://playwright.dev/docs/test-sharding)
- [Playwright docs: CI intro (GitHub Actions workflow, retention-days)](https://playwright.dev/docs/ci-intro)