THE TRUE COST OF END-TO-END TESTING IN-HOUSE

Developers on your team

Team size is a good indicator of your app’s testing footprint.

Factor in app age

App age

0 yrs

The age of an application affects the number of tests it needs due to factors such as codebase stability, accumulated complexity, and tech debt.

Coverage

Badge

Aiming for 80% E2E test coverage ensures critical paths are well-tested, significantly reducing the risk of undetected bugs.

Learn more

Time to create each test

0 hrs

2 hours per test case on average. This includes outlining, coding, creating test data, and verifying it works.

Time to investigate and fix a failed test

0 hrs / test

~1.5 hours on average. This includes re-running the test, reproducing the failure, debugging, PR reviews, and re-deploying the test code.

edit fail rate

Fail rate

High velocity teams with daily releases see about 7.5% of tests fail.

QA engineer salary

$0 / year

As of January 2022, the base salary for titles like Senior QA Engineer or Senior Test Automation Engineer in the US was $90,000 before benefits and other comp.

Number of SDETs

By default, we recommend the number of Software Development Engineers in Test (SDETs) based on industry standards.

An SDET enhances your QAE team by developing automated testing tools and frameworks, increasing testing efficiency, coverage, and reliability.

Set a custom SDET team size

SDET Salary

Include at least 1 SDET to set salary.

$0 / year

As of January 2022, the base salary for titles like Senior Software Engineer in Test and SDET in the US was $120,000 before benefits and other comp.

Run Cadence

You should run the suite at least daily, or more if you deploy continuously.

Run type

Tests run one after another on a single resource. This method is simpler and may have lower initial costs, but can be time-consuming, especially for large test suites.

Market rate per sequential run

The average cost per sequential test run across AWS (t3.micro, t3.small, t3.medium, m5.large), CircleCI, GitHub Actions, and Travis CI is approximately $0.03 per run.

Average test length

0min

3-5 minutes is a good benchmark

Tests run simultaneously across multiple resources. This method can reduce total execution time significantly but might incur higher costs due to the need for more resources.

Cost for parallelization of an individual test

Major vendors charge ~$130/test/mo for up to 25 parallel runs and ~$100/test/mo for 26 or more.

Number of tests ran in parallel per run

Max:

Example: If you have 200 tests, you might run 4 batches of 50.

Thank you! Your submission has been received!

Oops! Something went wrong while submitting the form.

What it takes

Tests for 80% coverage

Time to create

100

tests

Business days

Test runs

/ mo

(

[#tests]

tests

running

[per day]

)

100 test failures

100 test runs

Test failures

/ mo

Test maintenance

hrs / mo

QA team to hire

0 QAEs

0 SDETs

test

Team cost

/ year

1 QAE(s)

1 SDET(s)

With a team of 5 developers, we estimate you’d need 125 tests to reach 80% coverage. To run those tests 1x per day, while factoring in the cost of infra, it would cost you a grand total of $195,000/yr.

Running your tests [$type]

10 tests in parallel (45 batches)

Time spent

hrs / year

Run cost

/ year

Total cost of end-to-end testing in-house

10000

/ year

Compare with QA Wolf

THE TRUE COST OF END-TO-END TESTING IN-HOUSE

About QA Wolf

Resources

Legal

Hello!