Agents that operate phones and computers from screenshots alone, clicking coordinates the way a person would, fail in a specific way that current error checkers are poorly placed to catch. A step can look perfectly reasonable on the current screen but still be wrong because it contradicts something the agent saw several screens earlier. Examples include typing the price of the wrong item, filing under an account that was not selected earlier, or confirming a form after skipping a choice whose effect only shows up later. These are multi-hop errors, the same kind that sink chained question answering when an early "bridge" fact is wrong. They are also a clean instance of the gap in multimodal models between seeing things correctly and reasoning over what they saw. We propose to build training data for a step-level error checker (a "verifier") in a new way. We take correct demonstrations, deliberately substitute a controlled wrong action at a chosen step, and then actually execute that action in a live Android emulator, so every downstream screen is a real consequence rather than an edited image. Because we choose the error, we know three things for free: the first wrong step, which earlier screen held the evidence it violated, and how many steps pass before the mistake becomes visible.
The central claim is that verifier accuracy on screen-only agents falls off sharply as the distance to the violated evidence grows, and that existing checkers are blind to this axis. Those checkers are trained on organic failures, human binary labels, or static image edits. We claim a verifier trained on executed counterfactuals, labeled by that distance, flattens the curve, and that the gain transfers to organic failures made by other agents. Structural metadata (the accessibility tree) is used only as a privileged tool while generating the data. At test time the verifier sees pixels and action coordinates only.
What exists and what is missing
The closest work falls into three groups. The first group builds error checkers for GUI agents but obtains its errors expensively or uncontrollably. BacktrackAgent adds a judging module and a reflecting module, trained on judgment data curated from existing Mobile3M and Auto-UI execution traces. GUI-Shepherd trains a process reward model, meaning a model that scores each intermediate step, on 52,000 human-labeled step judgments with GPT-4o rationales. GUI-PRA avoids training altogether and prompts a judge with memory and parsing tools. None of these controls which kind of error the checker sees, and none knows where the violated evidence lives. We supply both.
The second group uses synthetic errors but in static form. LookAgain cuts target elements out of screenshots (its "CutPaste" strategy) to teach single-step refusal. Text-side methods such as REFINER, Text-in-Context, Hal-Eval and Masked Summarization corrupt fixed strings or captions. These methods supply our core technique of pairing each clean trace with a nearly identical corrupted one. They also constrain it, because static edits can leave artifacts that a model learns to spot instead of reasoning, and they cannot produce the real downstream screens that a wrong GUI action causes. Executing the error in the emulator is our answer to both problems.
The third group motivates the long-range focus. PAL-UI lets an agent pull back old screenshots, but its retrievals mostly stay within five steps. The multimodal reasoning literature (MATH-Vision, GeoSQA) finds that failures come from the reasoning done after correct perception, not from perception itself. The data-scale study On the Effects of Data Scale on UI Control Agents shows that clean demonstrations scale poorly to unseen apps. We read it as evidence that more data from successful runs will not teach an agent to recognize its own failures.
Research Program
The generator starts from successful trajectories on live AndroidWorld tasks and replays each one in the emulator up to a chosen step. At that step it substitutes an action drawn from one of four perturbation families.
The first family is a local misclick on a sibling element; the error is visible on the current screen. The other three are designed to be locally plausible:
- Bridge-value swaps. The agent types or selects a value that did appear earlier, but belonged to the wrong entity, for example the neighboring row's price.
- Precondition drops. A selection step is skipped, and its effect surfaces only later.
- Premature commits. A confirmation happens before a required check.
The accessibility tree is used only here, at generation time, to find siblings, earlier values, and preconditions. The generator then executes the rest of the trajectory under the original policy or under a scripted continuation and records every screen.
Each example carries three labels:
- the first wrong step;
- the evidence distance, meaning how many steps earlier the contradicted fact appeared on screen;
- the surfacing lag, meaning how many steps pass before the error is visible on screen.
Crossing evidence distance with surfacing lag gives a two-dimensional difficulty map. We expect local judges to succeed only in the region where evidence distance and surfacing lag are both small.
The verifier is a 7B vision-language model of the same backbone class as BTL-UI and PAL-UI. It takes the current screen, the candidate action, and past screenshots fetched with a PAL-UI-style look-back tool. It is trained to do three things: judge the step as correct or wrong, locate the first wrong step, and, as an auxiliary task, point to the index of the evidence screen. We use the verifier in two ways. It reranks N candidate actions from a fixed policy, and it triggers backtracking to the step before the flagged error.
For evaluation we hold out task templates entirely from generation. We also annotate about 300 organic failures, produced by two policies we did not train on, with first-error steps. This organic set is the test that matters, because synthetic-error work repeatedly finds that models trained on generated errors can master the generator's quirks and fail on real mistakes.
Baselines
The naive reference point is the base policy with no verifier, run on AndroidWorld with the same step budget. Published starting points are low. GUI-PRA reports InternVL3-8B at 6.03% success rate (SR), and GUI-Shift reports 6.0% Pass@1 before its training.
The strongest learned competitor is GUI-Shepherd's human-supervised process reward model. It reports a 40.5% AndroidWorld success rate when it guides reinforcement learning with the PPO algorithm, and a 5.1-point gain when used as an inference-time verifier. It is the strongest competitor because it is exactly what we aim to replace: a learned step checker, trained on expensive human labels, and used the same way we use ours. Its released labels may be unavailable. In that case we will train a verifier of the same backbone on an equal number of human-style binary labels, obtained by annotating organic rollouts. We will then compare both models under an identical reranking protocol, with the same policy, the same N, and the same tuning budget. Beating it means higher first-error localization on the organic test set and a higher AndroidWorld SR at equal N. It also means a clear margin in the long-distance cells of the difficulty map, where we predict the learned competitor degrades.
The strongest training-free competitor is GUI-PRA, reproduced with its memory and parser tools on a stronger backbone. It reports raising Qwen2.5-VL-7B from 38.04% to 57.61% SR on MobileMiniWoB++. BacktrackAgent's judge and reflector pipeline, which reports a 7.59% task-success gain over ReachAgent on Mobile3M, serves as the organic-error recovery baseline and is run on the same backtracking trigger.
Two ablations isolate the proposal's key ingredients. The first keeps our perturbation specifications but renders the downstream screens by editing screenshots rather than executing them. The second trains on executed errors from the local-misclick family only, which tests whether the long-range families are what matters.
Why not just…?
Why not just harvest organic failures from policy rollouts and label them by task outcome? This is the most serious threat. Organic failures are by definition on-distribution. Their drawbacks are that outcome checkers give labels for whole trajectories rather than a first wrong step, and that we expect organic failures to be dominated by local misclicks. We will measure the distance distribution of organic failures directly. We will also train an organic-only verifier with a matched example count, and an organic-plus-counterfactual hybrid. If organic-only matches us in the long-distance cells and on the organic test set, the synthetic generator is not worth its cost, and we will report that result.
Why not just prompt a strong VLM judge with full history or look-back, as GUI-PRA does? GUI-PRA itself reports that long histories cause "lost in the middle" failures, and PAL-UI's look-back seldom reaches past five steps. Neither result shows that a prompted judge can find evidence twenty screens back. The distance-stratified curve settles this directly. If the prompted judge's curve stays flat, training adds little on the long-range question.
Why not just edit screenshots instead of executing errors, which is cheaper and needs no emulator? Static edits cannot show the real consequences a wrong action causes several screens later. They also risk teaching the model to detect artifacts, a concern raised by Can Humans Dream of Electric Sheep? about model-generated errors. The rendered-versus-executed ablation, scored on the organic set, measures exactly this difference.
Why not just use the accessibility tree at test time? Desktop applications lack any such universal structure (OmniACT), and mobile trees are often incomplete (Screen Recognition). We therefore restrict the tree to data generation. We report as a limitation that generation quality drops on apps whose trees are poor.
Decisive Checks
The first check is the difficulty map itself. Every verifier is scored by first-error localization accuracy within each evidence-distance by surfacing-lag cell. The claim fails if the baselines do not decay with distance, or if our verifier decays just as steeply.
The second check targets shortcuts. We remove the history from our verifier's input. Its accuracy on long-distance errors should fall to chance; if it does not, the verifier is exploiting artifacts rather than reasoning over past screens.
The third check is transfer to organic errors, meaning localization on the 300 annotated organic failures from unseen policies.
The fourth check is downstream effect. We measure AndroidWorld SR under reranking and backtracking against GUI-Shepherd's reported +5.1 verifier gain. We also measure AndroidControl-High step accuracy against its reported +4.3 Type Match gain, and we report per-step latency, since BacktrackAgent's pipeline roughly halves inference speed.
For the cross-app multi-step benchmark GUI-Odyssey, there is no reported result found on Lacuna for verifier-based methods. We will report the base and verifier-augmented agents there as new numbers rather than as comparisons.
References
- BacktrackAgent: Enhancing GUI Agent with Error Detection and Backtracking Mechanism
- GUI-Shepherd: Reliable Process Reward and Verification for Long-Sequence GUI Tasks
- GUI-PRA: Process Reward Agent for GUI Tasks
- LookAgain: Closed-Loop GUI Grounding with Visually Grounded Reflection
- REFINER: Reasoning Feedback on Intermediate Representations
- Text-in-Context: Token-Level Error Detection for Table-to-Text Generation
- Hal-Eval: A Universal and Fine-grained Hallucination Evaluation Framework for Large Vision Language Models
- Masked Summarization to Generate Factually Inconsistent Summaries for Improved Factual Consistency Checking
- PAL-UI: Planning with Active Look-back for Vision-Based GUI Agents
- Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset
- GeoSQA: A Benchmark for Scenario-based Question Answering in the Geography Domain at High School Level
- On the Effects of Data Scale on UI Control Agents
- BTL-UI: Blink-Think-Link Reasoning Model for GUI Agent
- GUI-Shift: Enhancing VLM-Based GUI Agents through Self-supervised Reinforcement Learning
- Can Humans Dream of Electric Sheep? Human-Written Samples for Fine-Grained Vision-and-Language Hallucination Benchmarking
- OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist Autonomous Agents for Desktop and Web
- Screen Recognition: Creating Accessibility Metadata for Mobile Applications from Pixels