TurnSignal for Playwright: what it is, who it's for, what it costs
In short
TurnSignal is a hosted test-results service for Playwright Test (the Node.js test runner). A reporter in your CI sends results as tests run; TurnSignal keeps each test's history across runs and branches and tells you which failures are new (likely caused by the change), already failing on the default branch, flaky, or missing because a runner crashed.
Who it is for
- Built for: Teams running Playwright Test in CI who need history across runs: which failures are new, which tests are flaky and how often, and which results went missing. Teams whose tests are written or fixed by AI agents can also review those changes with it.
- Not needed for: A small suite where one run's HTML report answers every question, or single-run reporting only.
- Not a test generator: TurnSignal records and reviews tests; it does not write them.
What it tells you after every run
Every run opens with a plain verdict. Each test gets a label with the reason behind it:
- Failed: The last attempt of the test failed.
- New failure: Fails here but passed in its latest result on the default branch: most likely caused by this change.
- Also failing on main: Its latest result on the default branch also failed: not caused by this change.
- Known flaky: It was flaky in at least one of its previous 20 results (any branch). Check before blaming the change.
- No history: No default-branch result to compare against yet.
- Quarantined: Someone marked the test as known broken, with an owner and an end date. Its failures are not counted as new; CI still fails.
- No new failures: On the runs list: the run failed, but only in tests already failing on the default branch, known to be flaky or quarantined. Nothing new to fix in this change.
- Failing on main: Tests failing on the latest default-branch run, with how many runs in a row. Every branch shows them as "Also failing on main" until they are fixed.
- Flaky: Failed at least once, then passed on a retry or when the CI job was re-run.
- Missing: Planned for the run but never reported: the runner crashed, was killed (often out of memory), or the run stopped first. The reason is shown.
- Incomplete: A run where some tests never reported a result, or a machine stopped without finishing.
Key facts
- Setup:
npm i -D turnsignal, add['turnsignal']toreporterin playwright.config.ts next to existing reporters, and set the project token asTURNSIGNAL_TOKENin CI. - The reporter never changes Playwright's exit code. Results are written to disk in CI first;
npx turnsignal uploadsends anything that could not be delivered. - CI: GitHub Actions, GitLab CI, Jenkins, CircleCI, Azure Pipelines and Buildkite are detected automatically (shards merge into one run); any other CI works with
TURNSIGNAL_RUN_KEY. - The pull request comment (
npx turnsignal comment) works on GitHub only for now. Slack, Discord and signed webhook alerts work everywhere. - Also included: Playwright trace viewer, videos and screenshots in the browser; flaky tests over 7-90 days; report-only quarantine; read-only API and an MCP server (
npx turnsignal mcp). npx turnsignal governancereviews test changes in a pull request (weakened assertions, skipped tests, added waits) andnpx turnsignal analyzeflags weak tests; both run locally and upload nothing.- Secrets in errors and traces are redacted by pattern in CI before upload, and errors are checked again on the server; screenshots and videos can't be redacted and can be turned off.
- Not supported today: Playwright for Python, Java or .NET; Cypress.
Setup in three steps
- Install the reporter:
npm i -D turnsignal. - Add it next to your reporters in playwright.config.ts:
reporter: [['list'], ['turnsignal']]. - Sign in, create a project, and set its token as the
TURNSIGNAL_TOKENsecret in CI.
The docs have the setup for GitHub Actions, GitLab CI and other CI. Is it safe to add? See security and data.
Free during the public beta
TurnSignal is in public beta: every feature is free while we work with the first teams. No credit card.
- Each organization can record up to 3,000 runs and 300,000 test results a month, in up to 5 projects. Need more? email hello@turnsignal.ai and we'll raise it.
- Organization owners are warned at 80% and 100% of the limit: in the dashboard, and by email when we have their address. Over the limit, runs that already started still finish; new runs aren't recorded until the 1st of the next month (UTC), and your CI passes or fails exactly as it would without TurnSignal.
- Paid plans will come later. We'll announce them at least 30 days before anything changes. There's no card on file, so nothing can be charged automatically: you decide whether to move to a paid plan.
- Beta means some things will change as we learn from you. See the beta terms.
Questions people ask
What is TurnSignal?
TurnSignal is a hosted test-results service for Playwright Test (the Node.js test runner). A reporter in your CI sends results as tests run; TurnSignal keeps each test's history across runs and branches and tells you which failures are new (likely caused by the change), already failing on the default branch, flaky, or missing because a runner crashed.
Does TurnSignal replace the Playwright HTML report?
No. It runs as one more reporter next to the ones you have; the HTML report keeps working.
Can TurnSignal break my CI?
No. The reporter never changes Playwright's exit code, and if our API is unreachable your tests run as usual. Results stay on disk, and npx turnsignal upload sends them later.
Does TurnSignal need access to my repository or source code?
No. It receives test results from the reporter in CI, never your repository or source code. Test source files are left out of uploaded traces.
How much does TurnSignal cost?
It is free during the public beta, with a monthly limit of runs and test results per organization; no credit card. Paid plans will be announced at least 30 days before anything changes.
Which CI systems does it support?
GitHub Actions, GitLab CI, Jenkins, CircleCI, Azure Pipelines and Buildkite are detected automatically; any other CI works by setting TURNSIGNAL_RUN_KEY. The pull request comment works on GitHub only for now.
Does it work with Playwright for Python, Java or .NET, or with Cypress?
Not today. TurnSignal supports Playwright Test, the Node.js test runner (JavaScript and TypeScript).
How does TurnSignal decide a test is flaky?
A test is flaky in a run when it failed at least once and then passed on a retry or when the CI job was re-run. A failure is labelled known flaky when that test was flaky in at least one of its previous 20 results.
Can AI coding agents read TurnSignal results?
Yes. npx turnsignal mcp starts a read-only MCP server so an agent can ask which failures are new, which tests are flaky and what a test's history looks like.
Try it on your next run
Free during the beta. No credit card, no repository access.