Here’s why you’ll regret maintaining E2E regression tests someone else built for you

The concept is alluring, isn’t it? Instead of pulling developers off of feature work, or hiring a slew of new test automation engineers you can contract out test creation and hand over the automated tests for an in-house team to run and maintain. Why wouldn’t that work? Your team doesn’t have time to do the up-front work but they can handle some JavaScript if it’s served up on a platter. 

If hoping made it so. Alas, here’s what you’ll find when you try to maintain the E2E regression tests that someone else builds for you:

1. Building the tests is the easy part, maintaining them is 90% of the battle.
2. Hourly contractors like to rack up the hours.
3. If what they promise isn’t what they deliver, your team has to pick up the slack.

Let’s get into it.

◦ ◦ ◦

Expectation: Regression tests are static, like unit and integration tests
Reality: 50% of your test suite will need to be updated every 6 weeks

Unlike low-level tests that check individual functions that rarely change, end-to-end regression tests need to reflect the latest UI at all times. Any change to the front-end, requires a change to the test suite to keep it working. Research that we did shows that active development teams will break 50% of their end-to-end test suite every six weeks if they don’t stay on top of the maintenance. One of the biggest reasons that teams regret spending in the upper-five digits for a comprehensive test suite is that they assumed it would be a one-time cost, and not a lifetime investment in time and people.

◦ ◦ ◦

Expectation: You’ll only pay for what you need
Reality: Test coverage isn’t measured in hours worked

Typically, outsourced QA contractors work on an hourly basis. As we’ve written before, the issue is wildly misaligned incentives. Hourly contractors aren’t motivated to deliver quickly, and they can increase their margins by delegating work to less-experienced and lower-cost developers. 

What you’re looking for is high levels of test coverage, as quickly as possible, allowing your team to maintain or increase their velocity. What you’ll get is back-and-forth about test cases, administrative overhead, and tests of questionable reliability. 

◦ ◦ ◦

Expectation: Regression tests are plug-and-play with your CI/CD pipeline
Reality: Integration challenges and consulting-ware

Test creation shops will frequently use their own tooling for development, if alternatives aren’t well described in the contract with the client. While the vendor’s tooling may help them deliver faster, the client will have to adapt their systems to work with the new test suite. And this is an opportunity for vendors to lock in customers to long-term agreements. If the tests are written in a specific proprietary format or with proprietary extensions written by an external agency, you might be locked into that format indefinitely. If your teams decide to switch testing tools or frameworks, they could be in for a complex, time-consuming, and costly transition.

Two approaches that work better 

There are a lot of QA companies out there and with enough searching you may find the perfect vendor for your needs. But if you want to increase your chance of success — and by success, we mean stable, reliable, comprehensive test coverage well into the future — try either of these approaches:

On-site, performance-based contract hires. These are 6–12 month contracts with experienced and vetted QA engineers who work inside your organization. Having them be full-time positions means they can learn your people, systems, and ways of working. Likewise, your permanent team can see how the tests were constructed and learn how to maintain them over the long term. Create measurable performance-based compensation metrics (for instance, a complete test plan or automating all workflows in one section of the application) to help align expectations and costs.

QA as a Service. Providers like QA Wolf charge per test we manage, so our incentives are aligned with your desire for speed, and we’re available 24/5 just in case you need to talk with us outside of your normal business hours. We take on the maintenance of your tests so your team doesn’t have to build that expertise. We run our tests on our own infrastructure, so your website is all we need from you. And if, for some reason, you need to run the tests we wrote for you in-house, you can have them. They are all written in Playwright, one of the most popular open-source test automation frameworks available.

Let’s talk about your testing needs 

QA Wolf builds automated tests for 80% of a product’s workflows in less than four months — we’re happy to share our experience from more than 100 customers.