Did my PR break this test? New failures vs already broken on main

Why a red run on a pull request is ambiguous

Your pull request touched the checkout page and CI is red: five Playwright tests failed. The obvious reading is that your change broke five tests. Often it is not true. On a busy repository, some of those tests were already failing on main before you branched, one is a known flaky test that fails every few days, and maybe one is really yours.

Playwright's report can't tell these apart, because it only knows about this run. Its flaky label covers tests that failed and then passed on a retry within the same run; a test that fails on every attempt is simply failed, whether your change caused it or it has been broken on main since Tuesday.

The three kinds of failure

• New failure: the test fails here but passed in its latest result on the default branch. Most likely caused by this change. Fix it before merging.
• Already failing on main: its latest result on the default branch also failed. Not caused by this change. Someone should fix main, but it should not block your review.
• Known flaky: it was flaky recently (for example, it failed and then passed on a retry, or passed when the job was re-run). Check the error before blaming your change.

There is a fourth case worth naming: a test that has no history on the default branch yet, usually because it is new in this pull request. There is nothing to compare with, so treat it like a new failure.

Telling them apart by hand

1. Find the latest finished CI run on your default branch (on GitHub: the Actions tab, filtered by branch).
2. Open its Playwright report (download the HTML report artifact, or read the log) and look up each test that failed on your pull request.
3. Passed on main: a new failure. Failed on main: already broken. Flaky on main or in earlier runs: probably flaky.
4. Repeat for every failed test, and again after every push.

This works, and it is tedious enough that almost nobody does it consistently. The usual shortcut is to re-run the failed job. If it passes, the failure was flaky, or it was not, and nobody knows which.

Automating it: a verdict on every run

TurnSignal keeps every test’s results across runs and branches, so each failure on a pull request is compared with the default branch automatically. The run page opens with a plain verdict, for example: “2 tests failed. 1 new failure: passed on the latest main run, fails here. 1 failing test also failed on the latest main run, so not caused by this change.”

On GitHub, the same verdict is posted as a sticky pull request comment that is updated on every push instead of adding a new one. It groups failures into sections, in the order you should look at them:

• New failures: likely caused by this change, each with its file and line, the first line of the error, and links to open the trace, video or screenshot.
• Failures without history (new tests), Known flaky tests that failed, and Also failing on main.
• Missing: tests that never reported a result because a runner crashed or was killed, with the reason.
• Fixed since main and Slower than main (compared with the test’s usual time on the default branch).

Setting it up in GitHub Actions

Install the reporter and add it next to your existing reporters:

npm i -D turnsignal

// playwright.config.ts
reporter: [['list'], ['turnsignal']],

Sign in, create a project and store its token as a secret named TURNSIGNAL_TOKEN. In the job that runs Playwright (sharded or not), pass the token, and add a step that sends any results that could not be delivered:

- run: npx playwright test
  env:
    TURNSIGNAL_TOKEN: ${{ secrets.TURNSIGNAL_TOKEN }}

# Recovers results if a runner crashed or the API was unreachable.
- if: always()
  run: npx turnsignal upload
  env:
    TURNSIGNAL_TOKEN: ${{ secrets.TURNSIGNAL_TOKEN }}

Then add a final job, after all shards, that posts the comment:

report:
  needs: tests
  if: always()
  runs-on: ubuntu-latest
  permissions:
    pull-requests: write
  steps:
    - run: npx -y turnsignal comment
      env:
        TURNSIGNAL_TOKEN: ${{ secrets.TURNSIGNAL_TOKEN }}
        GITHUB_TOKEN: ${{ github.token }}

Shards of one workflow run are merged into one TurnSignal run automatically, so the comment covers the whole suite. The reporter never changes Playwright’s exit code: your CI passes or fails exactly as before.

What makes the comparison trustworthy

• Each failure is compared with the same test’s latest result on the default branch, not just the previous run of your branch.
• “Known flaky” is based on the test’s previous 20 results on any branch, and each label links to the runs behind it.
• Tests quarantined by your team (with a reason, an owner and an end date) are listed separately and not counted as new; CI still fails if they fail.
• Every count links to the tests behind it, and every label explains itself. See what the labels mean.