Teaching an Agent to Use a Phone: How We Built AI for Android and iOS Tests

A web app is a document, and the browser hands the whole thing over. Every button, field, and link is in the page’s HTML, along with the IDs and labels a test uses to find it. Every page has a URL, so a test can go straight to it. Playwright controls the browser directly, so a test can read anything on the page, click with a mouse, and type with a keyboard. Our agent clicks right where it sees a button in a screenshot of the page.

A native app is different on every count. It’s compiled code that draws its own screen, on a phone that sits somewhere else: an emulator, a real Android or iOS device. A test controls that phone remotely through Appium. The only view inside the app is its UI tree, the list of elements the app chooses to expose to the phone’s accessibility system, and some apps, like those built with Flutter or drawn on a canvas, expose almost nothing. There’s no address bar, so a test launches the app and starts wherever the app starts. Input is touch, not a mouse. And every phone model has its own screen size and pixel density, so a spot on one phone’s screen isn’t the same spot on the next.

Why the Agent Couldn’t Test Phone Apps

The agent was built around the browser, so on a phone, each part of how it worked broke:

• It couldn’t find elements: It got selectors by reading a page’s HTML, or by hovering over an element while our recorder captured it. A phone has no HTML and nothing to hover.
• It could only use a mouse: Its actions were click, scroll, drag, and keypress.
• It aimed at the wrong spot: A spot in a phone’s screenshot isn’t the same spot on the phone.
• Its tests would only work on one phone: A tap at a spot on the screen, or a comparison against a saved screenshot, only holds for the phone model it was made on.
• Its instructions were for the web: Everything it had been told about writing tests assumed Playwright and a browser.

The first thing the agent does on any screen is look at it, so that’s where the fixes start.

Seeing What’s on the Screen

Because QA Wolf's tests are ultimately written in deterministic code (so that the runs are fast, reproducible, and don't call a model on every step), we needed a way to find a selector for every element the test is going to interact with.

In a browser, the HTML and DOM provide the agent everything it needs to pick a selector. On a phone, the agent gets two partial views instead: A screenshot shows what a user sees but has no selectors, and the UI tree has selectors but doesn’t show which one belongs to which thing on screen.

We had to give the agent a way to marry up its two sources.

Now, when the agent spots the element it wants in the screenshot, it asks the phone which element sits at that spot. The phone answers with the element’s selectors, best first. The first one comes from the element’s own ID or label: on Android, its resource ID, or its content description if it has no ID, or its text if it has neither; on iOS, its name and type. If the element has none of these, the selectors lean on a nearby element that does, or on the element’s path in the tree.

const locators = {  addToBag: $("//*[@resource-id='add-to-bag']"),};

Selectors that differ by platform don’t have to mean two tests. Apps built with React Native, for example, give each element’s test ID to iOS as its accessibility ID and to Android as its resource ID, so a search field can be ~shop-search on iOS and //*[@resource-id='shop-search'] on Android. The agent keeps both in one screen object, a class that holds one screen’s selectors (the mobile version of a page object), and one test runs on both platforms.

Of course, knowing where a button is only takes you so far. Then the agent has to tap it.

Building Our Own Touch Actions

Initially, our agent used OpenAI's computer-use tool. It was built for a mouse and keyboard with the ability to click, double-click, drag, scroll, and type. But it could only reach web apps, so we'd have to build a custom tool to work with mobile apps and eventually desktop apps.

Rather than reinventing the wheel, we kept our browser actions shaped exactly like OpenAI’s, so what the model learned in training would carry over. But those shapes were made for a mouse, and translating them to a phone fell short in different ways:

1. A click can’t carry a selector. It says which mouse button to press and where, as an x and a y on the screenshot, and nothing else. Appium can find an element by its selector and tap it wherever it is, but a click had no way to ask for that. So even after the agent had found a button’s selector, it had to tap the button by its position instead, and it couldn’t try the step the way the test would run it.
2. Typing couldn’t replace text. Typing only adds to whatever field has focus. Appium can clear a field before typing into it, but typing had no way to ask for that. If a search box already said “shoes,” typing “boots” could leave “shoesboots.”
3. A swipe couldn’t set its speed. On a phone, speed changes what a swipe does. A slow swipe moves a list a few rows, and a fast one flings it, so it keeps scrolling after the finger lifts. The agent could only swipe with a mouse drag, which has no speed setting, so it couldn’t choose between a scroll and a fling, even though Appium can swipe at any speed.

So we built device-only actions, one for each gap: a tap that can take a selector as well as a spot, a fill that replaces a field’s text, and a swipe with a set duration, so the agent chooses between a scroll and a fling. The next challenge was to make sure that those actions were hitting the right parts of the app.

Aiming Taps at the Right Spot

Selectors aren’t always there. Apps built with Flutter or drawn on a canvas expose almost nothing, and while the agent explores a new screen, it taps what it sees before it has a selector for it. In both cases, it taps by position. But coordinates only work when the screenshot the agent is looking at perfectly matches the device. And it usually doesn't.

Take an iPhone 15 for example: a screenshot is 1,179 x 2,556 pixels, but iOS lays out tap zones in points — 3 pixels to a point for a 393 x 852 grid. On the other hand, Android takes taps in pixels. On top of that, model providers shrink large images before the model sees them (Anthropic above 1,568 pixels, OpenAI above 2,048), so the screenshot the agent reviewed wasn’t the size it thought it was.

So the agent was constantly missing its targets.

Telling the agent how to convert from screenshot to phone and back again didn't work because it had no reliable way to know what size image it was looking at. And serving smaller screenshots made them unreadable.

So in the end we took that job away from the agent. Screenshots are capped at a size the models don't shrink, and a separate job converts coordinates between that picture and the phone in both directions: taps, swipes, and typing at a spot on the way in, and the positions of elements the agent looks up on the way out.

Taps now land where the agent aims, on the phone it’s looking at. Of course, when the test runs on a different phone model, the same spot is somewhere else. You don't want one test per device — so how do you do it?

Making One Test Work on Every Phone

A tap at a given point on an iPhone 15 lands somewhere else on an iPhone 16 Plus. Because no one wants to manage separate tests for every device, most teams that do E2E testing for their apps pin their tests to one phone model and let the others go untested.

Not good enough for QA Wolf. We want every test to run on every device. So when a step has to tap by position, the test describes the target in words instead of giving its position, along with what the screen should show afterward:

import * as ai from "@qawolf/ai/ios"; // or "@qawolf/ai/android"

await ai.tap(this.driver, "the Add to bag button", {
  verify: "the button reads Added to bag",
});

Here’s what that line does:

• The first run on a phone model asks a model once. The model gets a fresh screenshot and the list of elements on the screen, and picks the button. The test taps it.
• verify checks that the tap did what it should. After the tap, a second model call checks that the screen now shows what verify describes. Only then does the test save the tap point and a small picture of the screen around it. If the check fails, the step fails and nothing is saved.
• Each phone gets its own saved tap. Taps are saved per phone model and OS version. An iPhone 15 and an iPhone 16 Plus each keep their own, so one test runs on all of them. A phone the test hasn’t seen before asks the model once, and every phone it has seen keeps its answer.
• Later runs replay with no model call. On every later run on that phone, the test compares the same spot with the saved picture. If they match, it taps the saved point, so the step is as fast and predictable as any other line of code.
• A changed screen fails the step, unless autoHeal is on. If the picture doesn’t match, because the app changed, the step fails and shows the saved picture, the new one, and the difference. With autoHeal: true, the model looks again, the test taps its new answer, runs verify, and saves the new tap only if the check passes.

Screens get the same treatment. A test can check that a whole screen looks right:

await ai
  .expect(this.driver)
  .toMatchScreenshot("cart", "the cart with 2 items and a Checkout button");

The first time, a model checks that the screen shows what the description says, and the test saves the screenshot as the baseline for that phone model and OS version. On a phone it hasn’t seen, a model compares the new screen with the baselines from other phones. It accepts differences that come from the device, like screen size, and rejects differences in content or state. After that, every run on that phone compares pixels with its own baseline, with no model call. If the app changes on purpose, autoHeal lets a model judge the new screen against the old baselines and replace the baseline when the change is expected.

That covers what the agent sees and how it touches a phone. The rest was teaching it how phones behave.

Teaching the Agent How Phones Behave

Besides what it sees and how it acts, the agent works from written instructions: notes on how to write a good test, which tools to use, and how to organize the code. All of ours were written for browsers, so phones needed a set of their own.

The phone instructions cover how to act on a touchscreen, how to pick selectors, how to explore an app, and how to organize a test’s code by screen. They also teach the agent the ways phones behave that browsers don’t:

• Don’t assume a fresh start: The app can pick up where the last run left off, so a test first gets to a known screen. One-time screens, like onboarding, are treated as optional: wait a moment, close it if it shows, and carry on.
• Wait for late pop-ups: A permission prompt can appear several seconds after whatever triggered it, so the agent waits and looks again before deciding it isn’t coming.
• Don’t trust an empty password field: Phones hide secure text from screenshots, so a filled password looks blank. The agent judges by what happens after it submits instead of typing the password again.

Writing the rules wasn’t the whole job. The agent chooses which instructions to read, and on a phone test it sometimes still chose the browser ones. So now the test chooses: when a test imports our iOS or Android library, the agent gets the phone instructions in place of the browser ones.

The agent also learns our phone library from the library’s own code instead of notes someone wrote by hand, so what it reads always matches what it calls.

Testing the Agent Itself

Every change to the agent’s instructions, prompts, or tools can make it better at one task and worse at another. So we test the agent the way we test apps: with evals.

We built nearly 50 mobile eval scenarios, each a task we give the agent on an Android emulator or iOS simulator, with a check of the result. They run against demo apps we built for the purpose: a React Native store, an airline app whose bookings outlive a reset of the app’s data, and a Flutter kiosk that exposes no UI tree at all, so every tap has to go by position.

The scenarios go from simple to complex:

• Simple: add one selector to a screen object, or check a hand-written selector before committing it.
• Medium: extend an Android-only screen object so an iOS test can use it, or tap through the kiosk with ai.tap.
• Complex: write a whole test from a one-line request, repair a checkout test after an app update changed every shipping field, or create a promo code in a web admin and apply it in the Android app.

Each scenario runs several times, and we track its pass rate. When we change the agent’s instructions, prompts, or skills, we run the scenarios the change could affect before it ships, and a model helps pick them from the change itself. That way, we catch a fix for one case that breaks another before it reaches customers.

What the Agent Can Do on a Phone Now

A phone took away almost everything the agent relied on in a browser: the page’s HTML, a mouse, an address bar, and one screen size. So we rebuilt each piece. The agent finds selectors by asking the phone what sits at a point in its screenshot. It acts through touch actions we built, because the computer-use tool it started with only knew a mouse. We handle the math between the screenshot and the phone, so its taps land where it aims. A model inside the test works out each new phone once, for taps and for screenshots, so one test can run on every phone model. The test’s import hands the agent the instructions written for phones. And a suite of evals checks that every change to the agent keeps it working.

The result is a test built on Appium, the open-source standard for phone testing, that the agent writes once and QA Wolf runs on emulators and real Android and iOS devices.