One of the hardest things to do with software development teams is increase and sustain velocity. Often, shipping more or shipping faster means increasing your team’s overall capacity by hiring additional developers. But with a recession looming and many companies freezing their hiring plans, savvy teams can look at other levers they have, like removing bottlenecks in the QA process. Here are five cost-effective changes you can make.
The problem with manual testing is that it’s time-consuming; and for the cost, it isn’t particularly good at catching bugs. Human error is unavoidable and computers were designed to do repetitive work like testing. Automating your end-to-end regression testing might be the single best investment you can make to increase velocity — and you’ll end up with a better quality product as a bonus.
Depending on the size of your application, a full manual regression suite could take days, sometimes weeks. If that work is being done by developers, that means they aren’t working on new features. And it’s not like their time is cheap, so why waste it on something a computer could do better and faster?
Now at least when developers are testing their own work, they can fix bugs as they go. But if designers, PMs, customer support staff or dedicated testers are running manual tests, you don’t even get that benefit. If developers aren’t getting timely and actionable bug reports during the development process, then bugs will get out and future sprints will be handcuffed by bug fixes.
If you decide to automate your end-to-end tests, there are two basic approaches, each with their own benefits and drawbacks.
First, there are coded tests that use a development framework. Microsoft Playwright (open source) is the newest and most feature-rich, and would be a better long-term choice than older frameworks like Selenium and Cypress. Although coded tests require technical skills to write and maintain, they’re necessary for more complex workflows.
Then there are the “no-code” or ”low-code” tools. Products like Rainforest QA or Ghost Inspector. These tools are designed for non-technical staff to create automated tests with a point-and-click interface. They’re an easy way to get started with automation, but you may find that they only work for simple workflows. You’ll also be locked into their proprietary vendor format.
With either approach — coded or no-code tests — your team will have to create and maintain them, as well as the infrastructure (more on that below). It’ll take a lot less time than doing manual regressions, but as we discuss next, our clients have seen that they get the most benefit by offloading that work to QA Wolf instead.
The second major velocity killer is having developers write and maintain automated end-to-end tests. Our internal data shows that a company needs 1–2 QA engineers for every 5 front-end developers to sustain high test coverage. In other words, each developer has to dedicate 20–40% of their time to get the same result without dedicated QA resources.
If you want to maximize the velocity of your scarce (and expensive) engineering resources, offload test creation and maintenance to experienced QA staff.
One option is to build an in-house team of QA engineers and SDETs (plus managers, potentially). An in-house team will help your developers focus on feature development and provide timely bug reports, but the costs can add up quickly when you include testing infrastructure on top of salaries. You can use our free In-house QA Cost Calculator to estimate the total cost for your company here, but remember that as your product grows, so will the test cases you need, and the people needed to maintain them.
A more cost effective approach is a partner like QA Wolf that offers Test Coverage as a Service. QA Wolf provides 80% end-to-end test coverage and ends up costing about half of what an in-house team would cost for the same coverage. As the largest provider of Test Coverage as a Service, each of our full-time QA engineers are able to manage 4–5 times as many end-to-end tests as an equally skilled in-house QA engineer. And with staff working 24/5 from the US, UK, and Australia, we’re able to investigate failures quicker and fix broken tests faster than the alternatives.
Automating your end-to-end regression tests will take your QA cycle from several days to a couple of hours and make a huge impact on your velocity. But if you have a large test suite, a team that deploys several times a day, or both, you might notice the automated tests are clogging up your deployment pipeline. This happens because most companies can only run 10–20 tests at a time. To ratchet up your velocity even further, you need to run all your tests at the same time. This is called parallelization.
Instead of hundreds or thousands of 5-minute tests going one after another over a couple of hours, you can run them in parallel and reduce the QA cycle to a few minutes. Your developers will also get faster feedback on their code, which means less downtime babysitting builds.
To do this you’ll need to make some pretty significant investments. One option is in-house infrastructure. We suggest a Kubernetes back-end to dynamically allocate resources, and at least one full-time person to manage it. We’ve gotten our system so efficient that we can run tens of thousands of test cases every day and provide our partners with unlimited, and fully parallelized test runs included in the base price of our service.
The other option is a service like BrowserStack to provide the infrastructure for you, but you may find that the cost to run your whole test suite is out of budget.
We like to say that flaky coverage is fake coverage because flaky tests create a lot of noise for developers who then waste time tracking down the real bugs in the code,slowing down the entire development process.
If you don’t know, a flaky test is one that sometimes passes and sometimes fails even though nothing within the application or the test itself has changed. Some flakiness is inevitable because of intermittent site issues like network hiccups or hard-to-reproduce race conditions, but when developers can’t get consistently accurate results, they have to fall back to slow, error-prone manual testing.
Sadly, flaky coverage is pretty common. Many of our clients tell us that they started with robust test coverage, but couldn’t maintain the tests and ship at the speed they wanted. As tests would flake out, or stop working altogether, they would simply be disabled (which, of course, would lead to bugs and slow down development anyway).
To increase velocity and keep it high, automated test suites need to be fast, but they also need to be free of false alarms. They need to point developers to real, verified bugs, and eliminate distractions.
QA Wolf effectively eliminates test flakes by triaging test failures 24 hours a day. Our tests are automatically re-run three times, and tests that fail are reviewed by our full-time QA engineers. Flaky or broken tests are fixed automatically, while verified bugs are passed along via Slack or to the client’s ticketing system (Jira, Linear, etc.).
What the client sees is a zero-flake test suite that only delivers human-verified bug reports, which lets developers focus on feature development and keeps team velocity as high as it can be.
Test automation is, hands down, one of the best ways to increase your team’s overall velocity, but getting the maximum impact is a major investment. Even before the recent economic downturn, building an in-house team and the necessary infrastructure was out of reach for many companies — 90% of companies have less than 50% test coverage.
QA Wolf has been changing the economics of test coverage so that even in today’s economic environment, you can maximize velocity and minimize risk.
For about half the cost of an in-house QA engineer, QA Wolf will scale your whole team to 80% test coverage in four months and continue to grow with your product as you build new features (which you’ll have a lot more time to do). You get all of the QA process optimizations described above — fully-automated tests, 100% parallelization, and zero-noise bug reports — 24 hours a day.
Tell us about your velocity goals at hello@qawolf.com and let’s see how QA Wolf can help.
QA Wolf is a hybrid platform & service that helps software teams ship better software faster by taking QA completely off their plate.
© 2024 QA Wolf, Inc.
