Playwright tests missing from CI results: crashed shards, OOM kills and timeouts

The symptom: fewer tests than you have

You split a suite across four machines with --shard=1/4 … --shard=4/4, merge the reports, and the result is green. But the suite has 1,200 tests and the report lists 900. Nobody notices, because a merged report shows what it received, not what it should have received.

This happens more as suites grow. More machines means more chances that one of them runs out of memory, hits the job time limit or loses its runner, and the bigger the shard, the more tests vanish with it.

Why a killed machine leaves no report

Playwright's reporters produce their final output at the end of a run. With sharding, the usual setup is the blob reporter on each machine and npx playwright merge-reports in a final job. A blob report is written when the run ends; if the whole machine or the Playwright process is killed first, there is no blob for that shard, and merge-reports merges the shards that did finish.

// playwright.config.ts
export default defineConfig({
  reporter: process.env.CI ? 'blob' : 'html',
});

The common ways a machine dies before Playwright can finish:

• Out of memory. Browsers are memory-hungry. When the machine runs out, the operating system kills a process, sometimes a browser, sometimes Playwright itself, or the CI provider stops the job.
• The CI job time limit. On GitHub Actions a job is cancelled after timeout-minutes (360 minutes unless you set it). A hanging test can eat the whole budget.
• Lost or preempted runners. Spot instances, autoscaled Kubernetes pods and self-hosted machines can disappear mid-run.
• A shell `timeout` or a killed container. Anything that sends SIGKILL gives Playwright no chance to write a report.

Stopped on purpose: interrupted and not run

Some stops are handled by Playwright, so they do show up in the report. With globalTimeout (a limit for the whole run) the run ends with status timedout. With --max-failures (or maxFailures in the config), Playwright stops after that many failures. Ctrl+C gives status interrupted. In each case the run itself says why it stopped, tests cut off mid-run get the interrupted status, and tests that never started are listed as "did not run".

That is useful: the report is honest about what happened. It is also a reason to read the totals and not only the colour, because a run stopped early can have no failures at all among the tests it did finish.

How to spot the gap yourself

1. List what should run: npx playwright test --list prints every test (and the total) without running anything. Run it with the same --project and --grep options as CI.
2. Count what reported: the merged report shows the number of tests it contains.
3. Compare the two on every run, and fail loudly (or at least warn) when they differ. A small script in the merge job is enough.
4. Record memory: log the machine's free memory during the run so an out-of-memory kill is not a mystery afterwards.

Keep results when a machine dies

The fix is to stop depending on the end of the run. A custom reporter receives onTestEnd for every test as it finishes; if it appends each result to a file right away, everything up to the moment of the crash is on disk. A later CI step with if: always() can then send it, as long as the machine or workspace survived. When the whole machine is gone, you still know which tests should have run, so you can list the ones that never reported.

Two settings reduce the damage in the first place: set timeout-minutes on the job a little above your normal run time, so a hang ends early instead of after six hours, and lower --workers on small runners, which is the most common fix for out-of-memory kills.

How TurnSignal handles it

TurnSignal's reporter writes every result to disk in CI before sending it, and at the start of each shard it records the list of tests that shard is going to run. When a shard ends or goes quiet, any planned test that never reported is shown as Missing, with the likely reason (for example, the machine stopped responding) and the lowest free memory seen on that machine. npx turnsignal upload in an always-run step (or the TurnSignal GitHub Action) sends whatever was left on disk, and running it twice is safe. For containers and pods, npx turnsignal run -- npx playwright test uploads before the container exits.

The reporter never changes your exit code, and it works next to your existing reporters, including blob and html. See the docs for setup.