Q: What’s worse, doing your taxes or waiting in line at the DMV?
A: Testing Salesforce.
Your grumbling testers aren’t imagining things. If a million things can go wrong when you test Salesforce manually, you increase the number of things that can go wrong when you automate by an order of magnitude.
But what makes Salesforce testing difficult is specific technical aspects, such as the shadow DOM, SOQL, or any of the vast number of customizations Salesforce affords. While those are certainly the contributing factors/elements, it's the way those and other aspects of the Salesforce tech stack affect planning, building, running, and maintaining the automated tests that we're going to discuss today.
The first problem that black-box testers encounter is the sheer number of test cases that need to be identified and created. Salesforce apps are dynamic. What a user (or test) sees on the screen is affected by all of the controls interacting with all of the profile types interacting with the various record types. The number of combinations (test cases) can quickly spiral into the thousands for even basic applications and integrations.
Most testers are familiar with three forms of compatibility testing: third-party integration testing, cross-browser testing, and mobile testing. Salesforce comes with a few additional integration concerns:
ISVs, those who are developing plugins for Salesforce, have an even larger compatibility testing surface and need to tack on these additional compatibility test cases:
Testers typically plan automation for any app by first manually executing each test step-by-step to capture all the variants of a given flow and ensure your suite has optimal coverage. In Salesforce, manual testing gets tedious quickly when accounting for all the variations and permutations, and tedium increases the risk of missed coverage.
While not a technical challenge per se, another source of missed coverage happens when developers use features like Flow Builder and Process Builder, which hide business logic deep within Salesforce’s underlying structure which obscures it from testers.
Building automated tests for Salesforce apps is uniquely challenging for three reasons: locators, framework limitations, and test data management.
The optimal locator in an automated test is a custom (and static) HTML element attribute, a “test ID,” so the test framework can always find what it is looking for regardless of changes to the DOM. Unfortunately, Salesforce dynamically generates the values for most IDs, so testers need to resort to other methods.
The second-best approach is to use CSS selector syntax. But, the significantly more complex DOM on any given Salesforce page challenges the limitations of CSS selectors.
This leaves us with the dreaded XPath option. XPaths are slower than CSS selectors, harder to read, and more brittle because they need to be tightly coupled to the structure of DOM. Any minor change to a page can cause an XPath to break. As you’ll read later, XPath’s damage isn’t limited to test creation
Salesforce pages frequently incorporate dynamic and asynchronous elements. That’s not exciting. It’s standard practice these days, but not everyone has the luxury of updating their crusty, old test framework on a whim.
Older versions of some frameworks over-complicate dynamic element support (we’re looking at you, Selenium) or the shadow DOM (let alone Salesforce’s custom implementation). However, some frameworks, such as Playwright, make interacting with shadow DOMs and dynamic elements practically effortless. Switching frameworks is no small effort, but might be worth the upgrade if you’re heavily reliant on Salesforce as part of your tech stack.
As with everything else, Salesforce makes an already difficult task, test data management, even more challenging. Their complex data model forces data integrity by running validation rules before all modifications, so testers can’t simply query the database for records and update them to match the requirements of the test. In Salesforce, such updates would either be prohibited or cause unintended side effects. Instead, most automators use subsetted, sanitized data regularly migrated from a production snapshot.
The problem with relying on production data for testing (apart from the obvious security concerns) is that you can’t predict whether the data you need to make a particular test pass will be available in any given snapshot.
Any approach you use to prepare production data snapshots for use in testing comes with a drawback:
The main issue with running automated e2e tests on Salesforce is the performance of the system under test (SUT) and the tests themselves.
As we said in the previous section, Salesforce frequently leaves us no choice but to write our locators in XPath. As if that wasn’t a big enough hit to our test performance, many Salesforce pages have a heavy DOM structure, so locators are often deeply nested. The more paths XPath has to traverse, the longer the element takes to find, the slower the test takes to run, the greater the risk of flakes, and the later the new feature goes out.
Most teams use Salesforce sandboxes and scratch orgs for testing. These environments are intentionally under-provisioned, which means that applications run slower, which means your tests take longer to complete.
Governor’s limits are restrictions on API calls, data storage, processing power, and other things that Salesforce uses to protect itself and its customers from the side effects of multi-tenant architecture when using Apex code. Salesforce also implements a separate set of API limits, which will throttle any tests that exceed them, likely causing the test to flake.
While governor’s limits generally prevent Salesforce’s customers from competing for resources amongst themselves when executing Apex code, understanding Salesforce is a world unto itself. Although remote, another org’s activity can degrade your application’s performance, causing delays and flakes. Particularly in Salesforce’s intentionally under-provisioned sandbox environments and scratch orgs.
Hopefully, it’s obvious that if you create a bunch of data on under-provisioned sandboxes or scratch orgs, your Salesforce application will slow down. There are other ways test data can negatively affect your test suite:
Unfortunately, due to the complexities of the Salesforce data model, it’s not always easy to design test data efficiently.
Because a Salesforce app has so many more moving parts than a standard web app, maintaining broken tests is absolutely Sisyphean.
Salesforce itself only has three major updates a year and does an excellent job of publicizing them but they’ve adopted a continuous deployment model, meaning their developers can release bug fixes, security updates, and performance enhancements at any time.
But the bigger risk is plugins from the AppExchange. The ISVs who develop those applications can (and do) ship updates at any time with very little advanced notice (if any). These updates can force your testers to drop everything else, potentially even a release you have in progress.
Remember how we told you that Salesforce gives automators limited locator choices due to its complex DOM and restrictions on adding custom elements to HTML attributes? Well, that also adds to the maintenance burden. XPath is a great tool, but it’s a drag to maintain convoluted XPath locators. Experienced testers on standard web apps know to use XPath as a last resort (for reasons mentioned in the section above). Unfortunately, when working with Salesforce apps, XPath is a first line of defense.
We hope this article can help you and your team communicate the challenges of automating Salesforce testing.
At QA Wolf, we specialize in building automated black-box tests for any application stack, and Salesforce apps are in our wheelhouse. What’s more, our offering spans the phases of test automation, so as long as we are running your tests, you can let us focus on creating, running, and maintaining your tests while you focus on delivery. Give us a shout.
QA Wolf is a hybrid platform & service that helps software teams ship better software faster by taking QA completely off their plate.
© 2024 QA Wolf, Inc.
