Computer-use agents that work only from screenshots, clicking and typing the way a person does, fail mostly through small errors that compound. A click lands a few pixels off, a menu takes a moment to open, a pop-up steals focus, and the agent keeps going on a false picture of the screen. When the screen fails to look as the agent expected, there are two different things that could have gone wrong. The agent may have misread the interface earlier, for instance by clicking the wrong icon two steps ago. Or the agent may have read it correctly while the world moved underneath it: the application lagged, ignored the input, or opened a dialog. These call for different fixes. The first means going back and re-choosing an earlier target. The second means waiting, dismissing, or retrying. Today's recovery modules mostly do not ask which one happened. They make a yes/no "did this step work?" judgment and undo the last step.
We propose to give a pixel-only agent an explicit diagnosis step. When an observation is surprising, the agent scores a small set of competing explanations of its recent history: "my click at step $i$ missed its target" for each of the last few steps, "the application did something I did not predict," or "nothing is wrong." Each explanation gets a graded cost, read directly from the model's internal scores instead of from generated text. The agent adopts the cheapest explanation and applies the recovery that matches it. The formal template comes from logic work on belief change, which unifies "I was wrong about the past" with "the world changed." The design choices about where the numbers come from are taken from information-retrieval work showing that graded scores read from hidden states beat yes/no labels and verbalized reasoning. The claim is that cause-specific, multi-step-back recovery with graded scores improves task success over one-step binary backtracking at matched compute.
Core Claim
The file supports three observations that, taken together, point to a specific mechanism.
First, recovery helps, but current recovery is binary and myopic. BacktrackAgent adds four modules to a mobile agent: a rule-based verifier that checks whether the screen changed, a vision-language judge that decides whether the new screen is an error, and a reflector that rolls back. It improves Mobile3M task success by 7.59% over the prior best agent. Its judgments are success/failure, though, and its rollbacks are local. It cannot represent "this step probably half-worked." It also cannot reach back several steps to a misgrounded click whose consequences only show up later.
Second, the logic literature already says how to choose between "revise the past" and "the world changed." Generalized Update represents beliefs as histories of states and transition events, each carrying a plausibility rank that measures how surprising it would be. When an observation conflicts with expectations, the agent adopts the least surprising complete history. Depending on which history wins, this either revises the initial assumption or infers an unobserved transition. Belief Change in the Context of Fallible Actions and Observations extends this so that both actions and observations can fail. A reliable present observation can then retroactively overturn the belief that a past action succeeded. That is exactly the "my earlier click missed" diagnosis. Belief Revision, Revised shows why keeping revision and update as separate operators breaks on alternating action/observation sequences, which is what a GUI trajectory is. These frameworks constrain our design in two ways. They were only ever run on toy examples, and they flag that ranking over full histories blows up combinatorially. We therefore bound the look-back window.
Third, the retrieval literature says where the ranks should come from. Beyond Polarization finds that pointwise rerankers which read a relevance score directly from hidden states beat ones that reason in text first. On DL19, Qwen3-32B scores 73.7 vs. 70.1 NDCG@10, and emitting the label after the reasoning instead of before drops the score from 72.7 to 68.2. RL training can make the reasoning model equally accurate at yes/no classification without closing the ranking gap. The paper's interpretation is that discrete text compresses a fine-grained signal. Choosing the least surprising explanation is a ranking problem over competing hypotheses. The prediction is therefore that a verbalized "what went wrong?" judge will be worse at choosing among explanations than a scoring head reading the same hidden states, even when both detect "something went wrong" equally well. Decomposed graded scoring as in Q-CARE also motivates scoring each hypothesis separately instead of judging the screen as a whole.
What is new is the combination. We make a pixel agent's recovery an explicit choice among cause hypotheses spanning several past steps, with graded costs from learned scoring heads. To our knowledge, none of the papers in the file does this.
Mechanism
At each step $i$ the agent stores four things: the screenshot, its intended target (a short description), the predicted coordinate, and the top few alternative candidate targets it considered. After action $t$ produces observation $o_t$, a lightweight surprise score compares $o_t$ with the agent's expectation. Only when the score exceeds a threshold does diagnosis run, which keeps overhead down given that BacktrackAgent reports roughly halved speed.
The hypothesis set over a window of the last $K$ steps contains three kinds of explanation:
- $h_{\text{act}}(i)$, for each step $i$ in the window: the click at step $i$ missed its intended target.
- $h_{\text{world}}$: the application did something unexpected, such as a lag, a pop-up, or a dropped input.
- $h_{\text{none}}$: nothing is wrong and the agent's expectation was off.
Each hypothesis $h$ is assigned a cost
$$C(h) = \sum_{i=t-K+1}^{t} c_{\text{act}}(i \mid h) + c_{\text{world}}(h) + c_{\text{obs}}(o_t \mid h),$$
where $c_{\text{act}}(i \mid h)$ is the cost of step $i$'s grounding having succeeded or failed as $h$ asserts, $c_{\text{world}}(h)$ is the cost of the unexpected application event that $h$ posits, and $c_{\text{obs}}(o_t \mid h)$ is the cost of seeing the current screen if $h$ were true. Each cost is a negative log-probability from a small calibrated head on the backbone's hidden states. Summing such costs corresponds to the sum-of-ranks rule in Hunter and Delgrande, with integer ranks replaced by continuous ones. The agent adopts $h^* = \arg\min_h C(h)$.
Recovery follows the diagnosis. Under $h_{\text{act}}(i)$, the agent returns to the state before step $i$ via back/undo, or by replaying the stored prefix, and grounds again using its next-best candidate. Under $h_{\text{world}}$, it waits and re-observes, dismisses the overlay, or retries the same action. If the top two costs are within a margin, it takes a cheap information-gathering action, such as waiting one beat and re-screenshotting, before committing.
The look-back window $K$ (we will test 1, 3 and 5) is the explicit answer to the tractability problem both logic papers raise. The hypothesis count grows linearly in $K$ because we posit at most one fault per window.
The metadata question enters only at training time. Supervising the heads requires knowing the true cause of each failure, and pixels alone do not reveal it. In virtual-machine environments such as OSWorld, we can read the accessibility tree after each action to check whether the click hit the intended element. We combine this with controlled fault injection: coordinate offsets that produce near-misses, artificial input delays, and adversarial pop-ups from PopupAttack. The result is labeled cause data. At test time the agent sees only pixels, so the structural metadata that is unreliable for acting (OmniACT documents that desktop platforms share no schema, and OSWorld that the trees reach millions of tokens) is used only as a source of training labels.
Research Program
We will build on an open pixel-only agent: an Aguvis-style model or the open agents in ScaleCUA. The cost heads are trained on fault-injected rollouts in OSWorld virtual machines and on Android rollouts in the AndroidInTheWild and Mobile3M settings, with environment infrastructure from ComputerRL. The backbone stays frozen in the main study, so every gain is attributable to diagnosis and recovery rather than better grounding.
The analysis has two levels. At the diagnosis level, we measure cause-attribution accuracy on held-out fault-injected episodes, broken down by fault type and by how many steps back the true fault lies. We also measure calibration of the costs. At the task level, we measure end-to-end success, the number of steps to recover, and wall-clock overhead.
Baselines
The naive reference is the forward-only base agent with no recovery, in the style of Aguvis. Aguvis reports 17.04% zero-shot success on OSWorld when trained only on web and mobile data. OSWorld itself reports 12.24% for its best accessibility-tree configuration (GPT-4) against 72.36% for humans. We will re-measure the base agent in our own harness rather than rely on these numbers. Beating it shows only that recovery helps at all.
A second simple reference is retry-or-back-one-step. On any detected surprise, the agent either repeats the action or presses back, chosen by a fixed rule. This tests whether diagnosis matters beyond "do something when surprised."
The strongest existing method is BacktrackAgent, reimplemented on our backbone with its rule verifier, vision-language judge, and reflector. Its reported gain is +7.59% task success over ReachAgent on Mobile3M. No absolute BacktrackAgent task-success value was found on Lacuna in the file, so the comparison must come from our rerun. We will run it on the same tasks, with the same backbone, the same cap on extra model calls and screenshots per episode, and the same tuning budget for thresholds. Beating it means higher task success on Mobile3M and OSWorld at equal or lower overhead. The gain should be concentrated on episodes whose true fault lies more than one step back, which is where its local rollback cannot reach.
Three ablations isolate the key ingredients. The verbal-diagnosis ablation keeps our hypothesis set and recovery policy but has the backbone pick the explanation by reasoning in text. This tests the hidden-state-scoring claim from Beyond Polarization. The binary ablation thresholds each hypothesis head to yes/no and picks any "yes," which tests whether graded costs matter. The $K=1$ ablation removes retrospection.
Why not just…?
Why not just let the agent reflect in text, ReAct-style, about what went wrong? This is the most serious alternative, because strong vision-language models can often name a misclick when asked. The evidence against it as the sole mechanism is indirect: in reranking, verbalized reasoning loses to direct scoring on fine-grained ranking even when yes/no accuracy is matched (Beyond Polarization). Whether that transfers to choosing among failure causes is an empirical question, and the verbal-diagnosis ablation settles it directly. If verbal diagnosis matches the heads on attribution accuracy and task success, we will report that the explicit hypothesis structure, not the scoring channel, carries the gain.
Why not just backtrack one step, as BacktrackAgent does? One-step rollback handles faults that surface immediately, and it may capture most of the benefit if most faults are like that. We will measure the distribution of fault-to-detection delay in OSWorld and Mobile3M rollouts. If delayed faults are rare, retrospection adds little, and the $K=1$ ablation will show it. The proposal's distinctive value then reduces to cause-specific recovery, wait versus re-ground, which BacktrackAgent's single rollback action also does not provide.
Why not just use a process reward model and search over actions? Step-wise progress rewards such as GUI-PRA, or continuous click rewards such as GUI-G², score how good a state or click is. They do not say why a state is bad, so a value-guided rollback cannot distinguish "wait for the lagging app" from "re-choose step $i$'s target." We will include a progress-score rollback variant at matched calls. If it equals our method, the diagnosis framing is unnecessary.
Why not just read the accessibility tree at test time? Where trees exist this would diagnose misclicks cheaply. But they reach millions of tokens on desktops (OSWorld), HTML pipelines cost 4,000 to 6,000+ tokens per step against about 1,196 for screenshots (Aguvis), and many applications expose none (OmniACT). We use the trees only as training labels and report a tree-at-test oracle as an upper bound on diagnosis.
Decisive Checks
The first check is diagnosis on fault-injected held-out episodes. The graded-head method should beat both the binary ablation and the verbal-diagnosis ablation on cause-attribution accuracy, with the margin growing as the true fault lies further back. If it does not beat the verbal ablation, the retrieval analogy fails for this setting.
The second check is end-to-end success on OSWorld (desktop) and Mobile3M (mobile), against reimplemented BacktrackAgent under matched extra-call budgets, reporting overhead alongside success. ScreenSpot accuracy should be unchanged, since grounding is frozen, which confirms that any gain comes from recovery.
The third check is natural rather than injected failures. We will hand-label causes on a sample of real failed OSWorld episodes and verify that the diagnosis accuracy measured on injected faults carries over. If it does not, fault injection is a poor proxy, and that is itself a reportable finding.
References
- BacktrackAgent: Enhancing GUI Agent with Error Detection and Backtracking Mechanism
- Generalized Update: Belief Change in Dynamic Settings
- Belief Change in the Context of Fallible Actions and Observations
- Belief Revision, Revised
- Beyond Polarization: The Generative Constraint of Chain-of-Thought in Pointwise Reranking
- Towards Query-Agnostic RAG Evaluation via Query Coverage and Claim Verifiability
- OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments
- OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist Autonomous Agents for Desktop and Web
- AndroidInTheWild: A Large-Scale Dataset For Android Device Control
- Aguvis: Unified Pure Vision Agents for Autonomous GUI Interaction
- GUI-PRA: Process Reward Agent for GUI Tasks
- GUI-G2: Gaussian Reward Modeling for GUI Grounding.