.svg){kind=logo}
Since the first outsourced QA vendors started offering automated end-to-end testing, they have billed by the hour. Some contracts may stipulate a flat fee for specific deliverables like test planning, but even those fees are derived from an hourly rate for labor.
This has several unfortunate consequences. One is that it obscures the true cost of running and maintaining a test suite by front-loading the cost of building an initial suite of tests. Another important point is that charging by the hour exponentially increases the cost of QA as the product grows and the team matures.
When it comes to automated end-to-end test suites, the total cost of ownership is driven by the amount of testing your team does, the number of tests in a test suite, and how frequently you run them. Over a five-year period, the high initial cost of building the core test suite ends up being a very small percentage of the total dollars spent.
Understanding this can help leaders project long-term costs more accurately, prevent budgets from constraining test coverage, and assess newer pricing models like pay-per-test more accurately.
To help understand this, we need to start with what goes into a single test.
While the lifecycle of a test is always the same, the amount of time spent on each phase varies considerably. That’s because not every test is the same — a log-in test may only take a few minutes to write and a few seconds to run, while a complex API test with multiple users could take several hours to create and 20 minutes to run.
But we’ve analyzed 3.5 million tests from more than 100 different apps and have come up with some pretty good averages for each phase:
As you can see, it takes almost as long to deal with a failing test as it does to create it in the first place. But unlike test creation, runs, failure investigation, maintenance, and bug reporting are recurring expenses that happen every time you want to ship.
With E2E testing, it’s death by a thousand cuts — little expenses adding up by the hundreds of thousands over time. While test creation is more or less a fixed cost, test runs, and test maintenance vary based on how frequently you decide to run your suite. The mistake lies in confusing the considerable one-time cost and the small but constantly increasing variable costs.
Let’s extrapolate what goes into a single test to an entire test suite managed by an hourly contractor.
As we’ve explained, it’s the costs incurred from running your tests that really drive the cost of ownership. But, because of the randomness of automated testing, the cost of a run is never the same.
You can play around with this on our Lifecycle of a Test page, but we’ve also developed a few scenarios to illustrate how much the 5-year cost of ownership is driven by the number of tests you have and the frequency with which you run them.
The values represent the annualized cost, starting at 250 tests four times per month at $65 an hour. Compute is calculated at .03 cents per minute. Calculations for this section use the same formula as our demo mentioned above to account for the randomness in failure rates, flake rates, and investigation, maintenance, and reporting times.
In the chart above, the size of the suite increases at a rate of 10% year over year, but we still run the tests only once per week. In an hourly pricing model, total costs rise modestly due to the labor involved in creating and maintaining a larger suite of tests. This increase is minimal compared to the cost impact of running these tests more frequently, as you will see in the following chart.
When you don’t add more tests but increase their frequency, adding one more run a week every year, the cost curve jumps much more. That’s because while creation costs stay flat, operational costs, specifically test maintenance and, to a lesser extent, the use of computing power for test execution infrastructure, increase. As you can see, with hourly billing, run cadence and not suite size that primarily inflates testing costs.
Under the hourly billing model, you’ll experience the most significant cost increases when you write more tests and run them more often. This scenario, common for those at any stage of the CI/CD growth continuum, leads to a sharp, steady, and almost exponential increase in computing power and maintenance expenses over time.
The per-test pricing model simplifies the economics of test automation. Under this model, vendors charge for each active test under management rather than the hours spent working on them. This model focuses on the output and quality of the work rather than how much time it takes to complete that work.
Under per-test pricing, you’re not just paying for the code. This model typically bundles the creation, execution, and maintenance of each test into a single, predictable cost. This can include:
The main benefit of the per-test model is that it reduces cost escalation over time compared to the hourly model. As you scale your operations, whether adding tests or increasing the execution frequency, the cost is transparent and manageable. Furthermore, the per-test pricing model incentivizes vendors to improve the efficiency and effectiveness of their testing practices by aligning the cost with the actual value delivered—each test executed. Under this model, the vendor assumes the cost of investigation and maintenance, so it is in their best interest to reduce those costs, which happens to align with the customer’s goals.
Let’s examine the two models using the same variables: a 250-test suite growing by 10% annually and an annual increase in run cadence once a week.
The black line illustrates the cumulative costs in the hourly billing model, which start at $120,000 in Year 1 and escalate sharply to just over $4.88 million by Year 5. This dramatic increase directly results from the hourly billing model, where costs rise as the testing suite grows and the test is executed frequently.
In contrast, the blue line represents the per-test pricing model. It starts at a similar initial cost in Year 1 but shows a much steadier and gradual increase in expenses over time, reaching around $732,960 by Year 5. This model effectively moderates cost growth by including all necessary test maintenance and execution within a fixed rate per test, avoiding the spikes associated with hourly rates.
So, while the per-test model may start similarly to hourly billing, its long-term financial impact is much more controlled and predictable.
QA Wolf is the only vendor on the market using a per-test-under-management model. Contact us and describe your specific situation. Our sales specialists can use our TCO calculator to provide a detailed forecast showing how per-test pricing can benefit your organization. This personalized approach will help you see how this pricing model can fit your unique testing requirements and financial goals.
QA Wolf is a hybrid platform & service that helps software teams ship better software faster by taking QA completely off their plate.
© 2024 QA Wolf, Inc.
