The game-like UI that makes Brilliant a great teaching tool, makes it a beast to test
Brilliant’s engineering leadership made 2024 a focus on quality. Continued success and continued growth were dependent on a smooth and delightful product experience for users and rapid development with minimal rework on the part of developers.
Automated testing was the obvious solution, and they started looking at different options (including an in-house team and LLM-powered vendors) that would meet their needs:
Handling the complexity of Brilliant’s UI
Brilliant’s app introduces a lot of complexity that most testers, and most QA tools aren’t able to handle. The lessons are delivered through interactive widgets with drag-and-drop functionality, unique animations triggered by the state of the UI, and responsive scaling for desktop and mobile devices. There are also dozens of permutations per lesson, like specialized hints or feedback triggered by particular errors. The whole system feels more like a game engine than a traditional application. As Brilliant evaluated their different options it became clear that most solutions weren’t built for game-like interactions or complex visual feedback loops.
In searching for a provider, it was really important to us that A) it was even capable of manipulating the browser in this way, because many of them are simply not capable of doing that; and B) have people who are able to think critically about a test’s purpose and goals.
—Jared Silver
Test coverage for legacy features with no documentation
When a company moves as fast as Brilliant, the functional requirements of older features aren’t always well documented. If the engineer who built it still works at the company, they’ve likely forgotten how the product was meant to work. And that creates a problem with tool-only vendors, and other outsourced QA solutions: someone from Brilliant needs to define the test cases
One of the reasons why QA has historically been a bit of a challenge for us is that functional product requirements for parts of the code base have been lost to time, and we would need to pull resources away from other things to reverse engineer more than a decade of workflows.
—Jared Silver
Release velocity and the impact on test maintenance
One of Brilliant’s primary needs was to allow developers and lesson designers to be able to ship without fear of breaking anything. This becomes extremely difficult when you’re trying to ship quickly, as every test needs to be revised alongside the product roadmap to prevent false positives. Before partnering with QA Wolf, the team at Brilliant relied on manual checks before and after releasing new code, which hurt their productivity and split their focus.
The cognitive overhead required by our developers and content producers to test every permutation, every time they published a new lesson or launched a new feature, definitely slowed us down. It would sometimes take an entire day for QA testers to go through the application.
—Jared Silver
.svg){kind=logo}
