You probably know some flaky people in your life. Unreliable and unpredictable people who cancel at the last minute or always show up late. They create chaos and force everyone else to adjust their plans.
That’s what flaky tests do to engineering teams. They force developers to stop what they’re doing and deal with tests that should be providing accurate and consistent results. Undo estimates that 620 million hours of developer time is wasted each year on failing tests, or $61 billion in developer salary.
Tests flake out for all sorts of reasons: Sometimes a PR will mess up a selector or change some part of the UI. Sometimes there’s a network hiccup. But there’s no way to tell from the red X in your pull request whether the test was flaking out or there's a legitimate bug.
A mid-sized product with comprehensive test coverage might have 500 test cases. Even if just 5% of those fail, that’s still 25 tests that the developer needs to investigate—running and re-running the tests trying to reproduce the failure to find the real bugs among all the noise. And, if necessary, also trying to fix any flaky tests that they find along the way.
It’s estimated that developers spend 20–30% of their time investigating and maintaining end-to-end tests. With engineering salaries starting at $200,000/year, debugging flaky tests could be costing tens of thousands of dollars in development time for each release.
Or, maybe the team is under-investing in test maintenance. Maybe it’s late in the sprint and the team doesn’t have time to fix their flaking test suite. We hate to admit it but we’ve all been there, and we know what happens next…
One of the biggest risks with flaky tests is the false sense of security they provide. Developers get used to seeing the red X on their PRs. They start to assume the issue is with the test and not their code. Or they get so tired of the same test blocking their deploy they just disable the test entirely.
That's when the bugs start getting through.
We all want to believe that our engineering team is different — that we will always fix flaky tests as soon as they’re discovered. And maybe you will! But is that even the best use of your development time? Developers should be focused on building features rather than maintaining complicated and brittle test cases.
Flaky tests create a lot of noise when developers should only be getting the signal. When teams use QA Wolf to create, run, and maintain their automated tests they’re guaranteed a simple Pass or Bug on every test run.
When tests fail, they are reviewed by full-time QA engineers in the US, UK, and Australia 24 hours a day. The human-verified bugs are passed on to developers, and the tests that need maintenance are updated automatically.
QA Wolf works with any web-based application, and automates any actions a user can take. We’ll get you to 80% test coverage in 4 months and add test cases as your product grows. With our in-house QA Engineers writing and maintaining tests your team won’t have to waste time investigating flaky tests — they can focus on building and shipping.
QA Wolf is a hybrid platform & service that helps software teams ship better software faster by taking QA completely off their plate.
© 2024 QA Wolf, Inc.
