Fix Flaky Playwright Tests Without Removing Assertions

Article · Measurement

A flaky Playwright test needs a reliable completion signal and the same behavioral checks it was meant to protect. Replacing a fixed sleep can help. Removing the assertion that catches wrong rows changes what the test protects.

Suppose an upload test fails now and then. The repair an agent proposes removes the assertion about duplicate rows and keeps the check that the page loaded.

The test turns green.

If the request was just “make this pass,” the patch comes close to doing what it was asked. The importer could now miss every duplicate and still pass the remaining check.

Before accepting the patch, the reviewer needs to know why the original assertion failed. Did it run too soon? Was the import broken? Did it look at the wrong element? Deleting it leaves all three questions unanswered.

Keep the failed run

A flaky test passes and fails without a relevant code change. That doesn't tell us the cause. Tests may share data, compete for resources, or check the page before it is ready. The application itself may also have a race that only appears sometimes.

Google's account of flaky tests describes how unreliable results make real failures harder to spot. Once people expect a check to fail randomly, they have another reason to dismiss the next failure.

For the upload, save the failed run's commit, input file, assertion, and trace if one is available. Look at what the page showed when the test failed. A successful rerun can help the investigation, but it doesn't explain the first result.

Ask the agent to investigate before asking it to edit. What did the test wait for? When does the application say the preview is ready? What would distinguish a test that runs too early from an importer that produces the wrong rows?

If the application announces completion before it has finished, the product may need a repair too. A longer sleep in the test would only make that defect harder to see.

Wait for a result you can observe

Suppose the test uses a fixed delay:

await page.getByLabel("Contacts CSV").setInputFiles("contacts.csv");
await page.getByRole("button", { name: "Preview import" }).click();
await page.waitForTimeout(500);
expect(await page.getByTestId("duplicate-row").count()).toBe(2);

It gives the preview half a second, then counts whatever rows exist at that moment. That may be enough on a quiet machine and too short on a busy runner. Raising the delay might hide the problem while making every successful run slower.

An assertion that waits for the expected result is more useful:

await page.getByLabel("Contacts CSV").setInputFiles("contacts.csv");
await page.getByRole("button", { name: "Preview import" }).click();
await expect(page.getByTestId("import-status")).toHaveText("Preview ready");
await expect(page.getByTestId("duplicate-row")).toHaveCount(2);

Playwright's retrying assertions check again until the condition matches or the timeout expires. Playwright also waits for a button to be ready before clicking it. That click doesn't tell us when an import finishes, so the test still needs to observe the preview's result.

In a real application, use its actual selectors and completion state. “Preview ready” must mean the preview has finished, rather than merely that the request has started.

Check which rows appeared

The example still checks duplicates, but a count alone is weak evidence. Two warnings might refer to the wrong rows. The preview could even repeat the same warning twice and satisfy the count.

The upload test should check which source rows were flagged and which first row they reference. Keep those behavior checks when fixing the timing. Replacing them with “the page loaded” would leave the importer almost entirely unchecked.

If each warning displays the source row and its original match, an assertion could look like this:

await expect(page.getByTestId("duplicate-row")).toHaveText([
  "Row 19 duplicates row 7",
  "Row 23 duplicates row 7",
]);

The wording is illustrative. Use the application's actual warning text, or assert the corresponding row fields if the preview presents them separately. The expected rows matter more than copying this particular selector.

The small browser fixture isolates those two requirements. Its button starts a preview that inserts the two warning rows after a controlled 700-millisecond delay, then announces completion. The test waits for the completion state and row text without adding a test sleep.

A second case deliberately renders row 23 as matching row 19. It still produces two warnings and announces completion. The check first confirms that wrong state exists, then requires the correct row-identity assertion to reject it. A count-only test would miss this defect. This is an explicit negative case, not an attempt to make a random race happen during every run.

The fixture uses fixed warning rows and page.setContent, documented in Playwright's page API. It does not parse a CSV, upload a file, or prove the application's whole importer works. Its purpose is to check the waiting and assertion behavior independently of those other components. The example pins Playwright, disables retries, and keeps failure traces; the recorded verification run passed both cases. That result concerns this bounded browser fixture, not the application's full importer.

Ask the agent to explain each assertion it changes. What mistake does the new assertion catch? If it removes an old check, where is that behavior covered now?

If the expected behavior itself is disputed, settle it before repairing the assertion. The trial-expiry example compares a generated test with an independent requirement.

A reviewer should be able to see that the repaired test still fails when the duplicate report is wrong. Using a recommended Playwright method is helpful; choosing the right expected result still requires thought.

If a test must be disabled, make that visible

Sometimes a team cannot fix an unreliable integration immediately. It may need to set the test aside while investigating. The cost is a gap in coverage, and that gap should be easy to find.

Link the failure, give someone responsibility for it, and decide when to revisit it. Record which behavior has lost protection. A test that quietly stops running can stay forgotten long after its original problem is fixed.

For this patch, ask for the failed trace, an explanation of what caused the failure, and assertions that still catch wrong duplicate rows. The CI result should belong to the revision containing the repair. Repeated runs can add confidence, though they cannot rule out every possible race.

The review can then focus on a concrete question: if the importer loses a duplicate tomorrow, will this test still catch it? Tolerating a known failing check is a separate workflow decision; it needs a visible owner and restoration condition rather than a weaker assertion.