We once spent a full quarter improving our automation framework. Better locators, better retry logic, better reporting, a genuinely nicer developer experience for writing new tests. Flakiness dropped. Everyone was pleased with themselves, myself included.
Then a test that verified refund calculations started failing three days after we shipped that ‘improvement’ quarter. Not because the framework regressed. Because someone had run a data-seeding script against the shared test environment, and a handful of orders our test relied on now had a different discount applied than they did last week.
The framework hadn’t broken. The ground underneath it had moved. And I realized we’d spent a quarter hardening the part of the system that was never actually our biggest source of instability.
We Optimize the Part That’s Fun to Optimize
Automation frameworks are satisfying to improve. There’s a clear before-and-after: fewer flaky retries, cleaner code, a nicer abstraction layer. Test data is none of that. It’s unglamorous: seed scripts, fixtures, environment resets, the tedious question of where a given test’s ‘user with three pending invoices’ actually comes from and whether it’s still in that state next week.
So it gets underinvested in, consistently, across teams that are otherwise pretty disciplined about everything else. I don’t think that’s a competence problem. I think test data is just less rewarding to work on, so it loses the prioritization fight against a shinier automation improvement almost every time, right up until it’s the reason a ‘stable’ suite starts failing for no visible reason.
The Failure Modes Are All the Same Shape
Once I started paying attention, nearly every ‘mystery’ flaky failure we had traced back to one of a small number of data problems, not a framework problem:
- A shared environment, mutated by someone else. Another team’s script, another test’s cleanup that didn’t run, a manual fix applied directly in staging, and now the record your test depends on doesn’t look like it did yesterday.
- State the test assumed but didn’t create. A test written against ‘a user with an active subscription’ that exists purely because someone happened to create one manually eighteen months ago, and nobody remembers that dependency until it’s gone.
- Data that doesn’t reflect production shape. Tests pass against a tidy, hand-crafted fixture and then the feature breaks on a real account that has, say, four hundred line items instead of three, a shape the fixture never represented.
- No realistic way to build large or complex datasets. So teams either don’t test at that scale at all, or they do it manually, rarely, right before a release, which is exactly when you have the least time to deal with what it finds.
None of these are automation bugs. They’re all data bugs wearing an automation costume, and no amount of retry logic or better locators fixes any of them.
Why This Gets Worse, Not Better, Over Time
The instinct once you notice this is to just ‘fix the data’: write better seed scripts, isolate environments, move on. That instinct is right, but it collides with a second problem: the data itself has gotten harder to work with, for reasons that have nothing to do with testing.
Realistic test data increasingly needs to resemble production, which means it increasingly touches the same categories of information that compliance and privacy rules exist to protect. Copying a slice of production into a test environment isn’t the shortcut it used to be. It’s a decision that now has to account for masking, retention rules, and who’s allowed to see what. Building large, realistic datasets synthetically instead is more work upfront, but it’s the only version of ‘realistic’ that doesn’t create a second, quieter problem the moment someone asks where the test data actually came from.
So the honest situation is: test data got harder at the same time as it got more important, and most teams’ tooling investment didn’t follow that curve. It stayed pointed at the framework.
What We Actually Changed
Nothing dramatic, and definitely nothing that felt as satisfying as the framework work did:
- Every test that needs a specific state creates that state itself, in its own setup, instead of depending on something that ‘should already be there’ in a shared environment
- Teardown runs even when the test fails, so a crashed run doesn’t leave a landmine for the next one
- Synthetic data generation replaced ad-hoc production copies for anything that needed to look realistic at scale, specifically so nobody had to make a judgment call about sensitive fields under deadline pressure
- We stopped treating the shared staging environment as a source of truth for anything a test’s correctness actually depended on. It’s fine for exploratory testing, and it’s the wrong foundation for anything automated and repeatable
The common thread: stop assuming the data will be there in the state you need, and stop assuming that state is safe to build by just copying something real.
Final Thought
If your ‘stable’ automation framework still produces failures nobody can quite explain, look at what the tests actually depend on before you look at the framework again. In my experience, the framework is rarely the bottleneck it gets blamed for.
The data underneath it is. It’s just less interesting to fix, which is exactly why it stays broken the longest.