A failing test is the easy case. It turns the run red, someone looks, and the problem gets a name.
The harder case is a test that never runs at all. It leaves no failure behind. The run simply has fewer tests in it, and fewer tests usually means fewer failures. The dashboard goes greener on exactly the night it should have raised a hand.
How tests go missing
None of these are exotic. Most teams with a large end-to-end suite have hit all of them:
- A setup project fails. Playwright skips every test that depends on it.
- A CI shard crashes or is cancelled. Its share of the suite is gone from the results.
- The run hits a time limit. Whatever had not started yet never will.
- A setup hook gives up. A shared test account could not be reset, so the test skips itself, and that skip looks the same as one you chose.
- A spec file is deleted or renamed by mistake. Its tests stop running, and nothing complains.
In each case, the report is accurate about what ran. It is silent about what did not. A pass rate of 98% on 400 tests reads very differently once you know the suite has 520.
Every run now knows what it expected
Donobu now records which tests each run was supposed to run, at the moment Playwright finds them. When the run ends, the Test Run page lists every expected test that never started. Once the run is over, those tests count as unexpected skips, not as silence.
There is nothing to set up. If your Playwright config uses defineConfig from @donobu/test and your CI sets a run ID, it works, including across shards.
The page also compares each run with the previous run of the same kind. If last night's nightly ran a test and tonight's did not, it appears under Missing Tests, whatever the reason.
A skip you meant, and a skip you didn't
Playwright records every skip the same way. A test you turned off on purpose and a test whose environment broke look identical.
Donobu now keeps them apart. Where your suite skips because something went wrong, mark it:
import { skipUnexpectedly } from '@donobu/test';
test.beforeEach(async ({}, testInfo) => {
if (!(await resetSharedDocument())) {
skipUnexpectedly(testInfo, 'Document reset timed out');
}
});
The Test Run page shows Unexpectedly Skipped in full color, and the skips you chose in muted gray. The run's totals include both, so a run's size is always the size of the suite, not the size of what happened to finish.
Failures that might just need another try
One more category used to hide in plain sight. When a new failure matches one a person already diagnosed as a timing or network problem, Donobu carries that verdict forward. Those tests used to land in Triaged, which reads as "handled".
They now wait in a Retryable section until a re-run decides. If the re-run passes, the test moves to Triaged. If it fails again, it moves back to Needs Triage, because this time it may be real.
What this means for your release decision
A test run is only useful if you can trust its denominator. With these changes, every Donobu test run answers four questions on one page:
- What failed, and why?
- What did we expect to run that never did?
- What was skipped by accident, not by choice?
- Which failures still need a second look?
A green run should mean the whole suite passed, not that part of it quietly went home early.
These features are live today for every Donobu account that sends test results to Donobu Cloud. If your team runs a large Playwright suite and wants to see what its runs are missing, book a walkthrough and we will show you on your own results.