Where does QA time really go?

(Hint: It’s not test creation)

We’ve analyzed 3.5 million automated test runs to show you what it really takes to build, run, and maintain an automated E2E test suite.

The data below comes from managing 350+ test suites (well over 100,000 automated tests) across 50+ industries from healthcare to fintech, dev tools, and even a couple of your favorite LLMs.

What goes into a single managed test

Test creation time

This one-time investment is how long it takes for a QA engineer to code and debug a single test in Playwright using the QA Wolf platform. Tests average 200 lines of code, though some may be 1000s of lines long.

Average test creation time

95 min

Test run time

The time it takes for a single test to execute on the QA Wolf infrastructure. Issues in the test environment can prolong the run time of a test suite. Babysitting the test run may fall to the developer or a QA resource.

Average test run time

9 min

Test failure rate

In any given run, some number of tests in a suite will fail due to flakes, bugs, or broken tests.

Failure Rate =
(# of tests) – (# of passes) / 100

Average failed test runs

15%

Investigation time

It takes on average 17 minutes per test to determine whether a failure is a flaky test (45%), a broken test (50%), or a bug (5%) .

Average investigation time

17 min

45%

50%