The half-life of an automated test suite is just six weeks

Automated end-to-end tests are, hands down, the best way to catch bugs and regressions before they reach users. But maintaining them is an expensive, time consuming pain. More than most companies can handle. In 2021, two-thirds of companies had less than 50% test coverage (SmartBear). Each release has the potential to break a selector or change the UI, which leaves developers waiting for the affected tests to be fixed before they can deploy. There are testing tools that claim to “self heal,” but product teams are usually changing more than those tools can handle. 

Exactly how much maintenance any test suite needs varies by the team’s velocity and output, but as one of the largest providers of automated test coverage (test development, runs, triage, and maintenance) we can get an average rate of decay for several high-performing teams. 

Research question

How quickly will test coverage decay if all test maintenance stops but developers keep shipping new code?

Methodology

In the name of science, we created an experimental test suite using a random sample of 50 end-to-end tests from 20 different clients. The tests ran against the client’s actual testing or staging environments every day for five months. But unlike the “real” tests, our suite of duplicates didn’t get fixed as our clients released new features. 

Automated end-to-end test suites lose 5–10% of their coverage each week without maintenance

As you can see, the experimental test suite started to break down almost immediately. We lost 25% of our test coverage in the first week, and it just kept dropping from there. Of course, the client’s working tests were repaired within 24 hours, but our experimental test suite was left to linger. 

As the weeks went on, the clients continued to ship new features and make changes to their UI. As they did, the automated tests needed constant attention. By week six of our experiment, half of the test coverage we started with was gone. And by week 12, only a dozen tests — barely 25% — were still working. 

You might be wondering why the graph shows so many ups and downs, instead of a nice, smooth line as the tests broke. That’s because automated tests are fickle — they can break one day and work the next. In some cases, a bug may have caused a test to fail but it was fixed the next day. In other cases, a temporary network issue may have caused a one-off flake. 

Still, the downward trend is clear: Without someone constantly maintaining the test suite as the product changes, you’ll lose 5–10% of your test coverage per week.

QA Wolf keeps up with product teams

For comparison, here’s what actual test coverage looked like over the same period. Tests would still fail from time to time — that’s just life with E2E tests — but they were maintained 24 hours a day by dedicated QA Wolves. Aggregate test coverage averaged 98% and never dropped below 92%. 

Product engineering is a race to market. QA engineering is a race to keep up. 

In a relay race, the slower runner sets the pace for the whole team. In the world of software development, that’s usually QA, which has to:

1. Create new tests as the product grows; while
2. Triaging tests every time there’s a failure; and
3. Filing bug reports, re-running flaky tests; or 
4. Repairing existing tests that were broken by the latest release

It’s an uphill battle and it never stops, never slows down, and only becomes more challenging the bigger the application becomes. Which is why an experienced QA engineer can only handle 30 or so tests on their own. After that, the maintenance burden becomes too big and the QA team needs to expand to meet the needs of the product.

That’s why high-performing teams like Chief and Possible Finance choose to work with QA Wolf. By providing 24-hour test triage and maintenance, we can keep pace with their developers and let them move fast without breaking things. 

QA Wolf works with any web-based application, and automates any actions a user can take. We’ll get you to 80% test coverage in four months and add tests as your product grows. You’ll be able to focus on building and shipping, and not have to worry about maintaining a test suite because someone from QA Wolf is doing that for you. 

“QA Wolf frees up your in-house engineers and team members to be able to focus on product work and development. I like that I can lay out what needs to be automated, and QA Wolf will go ahead and automate it. It allows your team to focus on work and making products better, while also knowing that you’re going to receive confidence from an automated test suite.”
‍—Whitney Sanborn, Lead Test Engineer @ Chief

‍