The suite that cried wolf

TL;DR:

• A suite nobody trusts is a credibility problem, not a tooling problem.
• Migrating to a shiny new framework rarely fixes it. It just moves the mess somewhere newer.
• Three usual suspects: flaky waits, shared state, and tests that assert too much at once.
• To find the real cause, follow the click: know exactly which frontend and backend code runs, and in what order.
• Trust comes back the boring way: one honest green run at a time. I'm still on that road myself.

The Monday nobody believes the red build

Every team I've joined has had one. A pipeline that turns red, and instead of anyone looking at it, someone just... reruns it.

I'll be honest: I've been that someone. More times than I can count. Not out of laziness, but because experience had taught me the red was probably lying. I still catch myself reaching for the rerun button now and then. Just much, much less than I used to.

That's the moment a suite stops being a safety net and starts being background noise. Nobody opens the report. Nobody blocks a release on it. It still runs, dutifully, every single build. It's just not believed anymore.

I've been thinking about this a lot while building the desktop automation framework I've been writing about in this series. Before I write a single assertion, I want to know: when this test fails six months from now, will anyone actually trust what it's telling them?

Changing tools moves the furniture, not the mess

The instinct, when a suite cries wolf too often, is to blame the tool. "Selenium is flaky, let's move to Playwright." "WinAppDriver is unreliable, let's try something newer." Sometimes that's genuinely true. Most of the time, it's a very expensive way to avoid a smaller, less exciting conversation.

I've been on teams that spent a quarter migrating frameworks and landed, almost exactly, back where we started: a suite that's green on good days and mysteriously red on bad ones, just written in a different syntax now.

A new tool doesn't fix why a suite lost its credibility. It just gives the same problems a fresh coat of paint, and a few quiet months before we all remember why we stopped trusting it in the first place.

The usual suspects

Before you touch the framework, go looking for these three, because in my experience it's almost always one of them:

• Flaky waits. Hardcoded sleeps, "wait and hope" patterns, or waiting for the wrong signal entirely. The test isn't wrong, it's just impatient.
• Shared state. Tests that only pass in a specific order, or that quietly depend on data another test left behind. Run them alone and they fail. Run the whole suite and they "work."
• Tests that assert too much. One test checking five unrelated things. When it fails, nobody knows which of the five actually broke, so the failure gets dismissed as "probably nothing."

Every one of these produces the exact same symptom: a red result that doesn't mean what it says. And once that happens a few times, we stop reading the message at all.

Even Google wrestled with this. Back in 2016, ten years ago now, they reported that about 84% of the transitions from pass to fail in their test corpus involved a flaky test (Flaky Tests at Google and How We Mitigate Them, Google Testing Blog). The number is old. The problem clearly isn't: a decade of new tools later, we're all still having the same conversation.

A test that fails for the wrong reason often does more damage than a bug that was never caught in the first place.

Follow the click all the way down

Spotting the suspect is half the job. Proving it means understanding what the test actually exercises, not just what it clicks.

Take a UI test that clicks a "Save" button and sometimes fails. Before touching the test, I want to know what happens behind the curtain:

• Which frontend function does that click trigger?
• Does it call the backend once, or kick off a chain: a validation request, then the save, then a refresh of the list?
• In what order do those calls happen, and does the UI wait for all of them, or only the first?
• What does the backend do with the request: write to a database, put a message on a queue, call another service?

Once you can draw that chain, the flakiness usually stops being mysterious. The test waits for the button to re-enable, but the list refreshes from a second, slower call. Or the backend saves asynchronously, and the next step reads data that isn't there yet. The test isn't randomly wrong. It's checking at a moment the app never promised to be ready.

So read the frontend handler, skim the backend endpoint, open the network tab, ask the developer who wrote it. It feels slower than adding another wait. It's the only thing that actually narrows the cause down.

Rebuilding trust, one boring run at a time

There's no shortcut here, and I'd be lying if I said otherwise. Trust comes back the slow way: a test fails, someone investigates, and the failure turns out to be real. Do that enough times and the team, me included, starts looking at red builds again instead of reflexively rerunning them.

That's genuinely the bar for the framework I'm building right now. Not "does it run fast," not "does it look modern." Just: when it says something broke, is it telling the truth?

If your suite can't clear that bar yet, a new tool won't get you there. A diagnosis will.

And a confession, since this is a journal: my own suite doesn't clear that bar yet either. There are a few tests in it that have been on the "rework this" list for longer than I'd like to admit, and I'm not proud of that. But I'm working on them right now, in small steps, one test at a time: follow the click, fix the cause, move on to the next.

So if you have a list like that too, consider this an invitation: join me in the fight against flaky tests. Pick one this week. Just one.

Next up: once the suite is honest again, the next question is where each test should actually live. That's Episode 3.