Can you trust AI-written Playwright tests? How to review what agents change

Why AI-written tests need a different review

More and more Playwright tests are written by coding agents such as Claude Code, Cursor and Copilot, and Playwright itself ships three test agents: a planner that explores the app and writes a Markdown test plan, a generator that turns the plan into test files, and a healer that runs failing tests and repairs them.

That speed changes where the risk is. When a person fixes a failing test, a reviewer usually knows why it failed. When an agent fixes twenty tests in one pull request, the diff is long, every test is green, and the easiest way to make a failing test pass is to make it check less. Nobody needs bad intent for this to happen: the agent is asked to make the test pass, and it does.

What the Playwright healer does

According to Playwright's documentation, when a test fails the healer replays the failing steps, inspects the current UI to find equivalent elements or flows, suggests a patch (for example a locator update, a wait adjustment or a data fix), and re-runs the test until it passes or its guardrails stop the loop. Its output is a passing test, or a skipped test if it believes the functionality itself is broken.

Most of those patches are exactly right: a button was renamed, a locator changed. But "a wait adjustment" and "the functionality is broken, so skip it" are also the two most common ways a real bug slips through. The review question is always the same: does this test still fail when the feature breaks?

Changes that weaken a test

These are the patterns worth stopping on in any AI-authored test diff, from most to least serious:

• The test is disabled or deleted: test.skip(), test.fixme(), or the test removed. The suite gets greener and the feature is no longer tested.
• An assertion is weakened: toHaveText('Order #1042 confirmed') becomes toBeVisible(); expect becomes expect.soft; an assertion is removed or negated. The test now passes whatever the page says.
• The expected value changed: toHaveText('$42.00') becomes toHaveText('$44.10'). Sometimes the product really changed. Sometimes the test was updated to match a bug. Check whether application code changed in the same pull request.
• A wait, timeout or force was added: page.waitForTimeout(), a longer timeout, test.slow() or click({ force: true }). These can hide a real slowdown or a real overlay covering the button.
• A snapshot baseline was updated: a new screenshot baseline accepts whatever the page looks like today.
• A new test asserts nothing, or does not await its assertion: it passes as long as the page loads.

And the changes that are usually safe: only a locator changed, or the test was renamed and its steps and assertions are the same.

A review checklist for AI-authored test changes

1. For every changed test, compare its assertions before and after. Fewer, softer or negated assertions need a reason in the pull request.
2. If an expected value changed, find the application change that explains it. No application change is a red flag.
3. Treat new skips and fixmes like a deleted test: who owns it and when does it come back?
4. Question every new fixed wait and longer timeout; ask what the test is waiting for and use a web-first assertion instead.
5. For new tests, check that each one ends with a meaningful assertion about the result the user cares about.
6. Know who made the change: a person, an agent, a bot, or a person with an AI assistant (Co-authored-by trailers).

Automating the first pass in CI

Reading every assertion in a long AI-generated diff is exactly the work people skip. npx turnsignal governance (in the free turnsignal npm package, 0.6.0+) does the first pass in your pull request job. It compares every changed test with the base branch and sorts each change into Needs review (assertion weakened, expected value changed, test disabled or deleted), Worth a look (sleep, longer timeout, force: true or test.slow() added; snapshot updated), Information (new tests, with static checks such as "no assertion") or Safe (locator-only change or rename).

Each change says who made it when git knows: a person, an agent, a bot, or AI-assisted. It runs with git on your CI machine, needs no TurnSignal account or token, and uploads nothing. On GitHub it posts one sticky comment on the pull request:

test-changes:
  if: github.event_name == 'pull_request'
  runs-on: ubuntu-latest
  permissions:
    contents: read
    pull-requests: write
  steps:
    - uses: actions/checkout@v4
      with:
        fetch-depth: 0              # needs the base branch history
    - run: npx -y turnsignal governance
      env:
        GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

By default the job only reports and always passes. When you trust its judgement on your codebase, add --strict to make it a required check that fails when a change needs review. On other CI, or on your laptop, npx turnsignal governance --base origin/main prints the same report. For tests that already exist, npx turnsignal analyze flags weak patterns such as tests without assertions, un-awaited assertions and hard waits, also locally.

The bigger picture

Agents will keep writing more of the suite. The teams that benefit are the ones that keep an independent check on what those tests actually verify, and a history of how each test behaves in CI over time. TurnSignal is built for both: a verdict on every run, flaky tests with evidence, and a review of every test an agent changes. See the docs for setup.