Open-weight computer-use agents are models that operate a computer or phone by looking at screenshots and emitting clicks and keystrokes. On standard benchmarks they now match or beat proprietary systems at locating screen elements. Those benchmarks, and most of the training data behind them, are built from clean screens. The data is extracted automatically from the accessibility tree, the structured list of on-screen elements that operating systems expose for assistive software, and it has no pop-ups, notification banners, stray background windows, or look-alike buttons. Real screens are full of all of these. This proposal asks whether open-model parity survives realistic visual clutter on desktop, web, and mobile. It also tests a specific way to make it survive: train the agent on distractors chosen because they fool that particular agent, not on random clutter pasted onto screenshots.
The bet rests on a finding from error recovery. Recovering Policy-Induced Errors showed that training on the mistakes an agent actually makes works far better than training on randomly injected errors, because each agent's failures follow its own distribution. We carry that idea from the agent's internal mistakes to external screen noise. If an agent's susceptibility to distractors is just as idiosyncratic, then synthetic distractors found by searching against the agent should transfer to real cluttered screens better than random synthetic distractors of the same quantity. The comparison is the point: if random clutter works just as well, that is a useful negative result and the proposal reduces to a benchmark.
Core Claim
Current parity results, such as ScaleCUA-32B's 94.4% on MMBench-GUI L1-Hard against GPT-4o's 53.5% in ScaleCUA, and OS-Atlas-7B's 82.47% on ScreenSpot in OS-ATLAS, are measured on the same clean distribution the training data comes from. We claim three things. First, these models lose much more accuracy on genuinely cluttered screens than the clean numbers suggest, and their errors cluster on target-like distractors. Second, training on distractors mined against the policy closes more of that gap on real clutter than training on random overlays, at matched data and compute. Third, distractor robustness learned on dense desktop screens transfers to mobile but not in the reverse direction. This third prediction follows from the transfer asymmetry already reported: desktop-to-mobile transfer is positive in ScaleCUA, while mobile-trained GUI-Shift fails on desktop ScreenSpot-Pro (GUI-Shift).
How This Differs From Prior Work
Several papers motivate the gap. ScaleCUA and OS-ATLAS build cross-platform open agents from accessibility-tree-synthesized data, and neither evaluates clutter. OSWorld documents that real desktop observations overwhelm agents, but it measures this through accessibility-tree length, not visual distraction. Text retrieval work shows the same failure in another modality. Improving Zero-shot Reader by Reducing Distractions finds that readers degrade as irrelevant documents accumulate and fixes this by allowing abstention. CompAct fixes it by compressing context. Both are text-only. We borrow the abstention idea as a "dismiss or report occluded" action.
The closest GUI work on visual noise is Visual Test-time Scaling for GUI Agent Grounding (RegionFocus). It zooms into sub-regions at inference time without training and cannot help when the target is hidden behind a modal. We change the model's training instead, and we explicitly target the occluded case.
Two papers supply our methods. RoTS's fragility-driven exploration provides the idea of searching against the policy; we apply it to screen composition, not trajectory branching, which is also far cheaper than its 800k-sample tree synthesis. GUI-Shift provides the training signal: reinforcement learning with a rule-based reward that credits any click inside the target box.
Two findings constrain the design. RoTS found that more than about 10% reflection data makes agents "over-reflect," so we expect the same risk of over-dismissal and will sweep the distractor ratio. ScaleCUA found that GUI-specific fine-tuning erodes general multimodal skill unless general data is mixed in, so we keep its general-data mixture fixed.
Research Program
The first component is a cross-platform clutter benchmark with two parts.
The synthetic part takes held-out screenshots from the OS-Atlas corpus and the ScaleCUA-Data corpus across Windows, macOS, Linux, Android, and web. It composites four families of distractors onto them, with ground truth from the original accessibility trees:
- copies of the target with near-synonym labels or the same icon class;
- modal dialogs that partly or fully cover the target;
- notification and ad banners;
- background windows.
The real part matters for the synthetic-to-real question. It consists of screenshots captured from live OSWorld Ubuntu VMs, AndroidWorld emulators, and MacArena sessions with system notifications, update prompts, cookie banners, and multiple open windows left on. Human annotators label targets and occluders. No synthetic distractor is ever evaluated as if it were real.
We cross each screen with two instruction styles. One is the programmatic element label. The other is a colloquial reference written by annotators, in the spirit of the "letters are too small" versus "Font size and style" gap in Intuitive Access to Smartphone Settings. This lets us test whether visual clutter and natural phrasing compound.
The second component is the mining procedure. For a fixed base policy, such as ScaleCUA-7B, we search over distractor family, position, scale, and label text for each training screen. We keep the configurations that move the policy's click outside the target box or onto the distractor. Labels are drawn from the target's own accessibility-tree neighbors and from paraphrases. When an occluder fully hides the target, the correct action becomes dismissing it or abstaining. We train with reinforcement learning using a box-hit reward and a dismissal reward, and we re-mine against the updated policy every round so the distractors stay matched to its current weaknesses.
Baselines
The do-nothing reference is the unmodified base policy, ScaleCUA-7B and ScaleCUA-32B from the released OpenGVLab/ScaleCUA code, together with OS-Atlas-7B. All are evaluated on clean and cluttered splits. Their gap is the headline measurement.
The strongest existing method for visual noise is RegionFocus applied to the same base models. In its paper it lifts Qwen2.5-VL-72B to 61.6% grounding accuracy on ScreenSpot-Pro. For full-system comparison on high-resolution screens, ScreenSeekeR reports 48.1% on ScreenSpot-Pro (ScreenSpot-Pro). There is no reported result on Lacuna for any method on cluttered or occluded GUI screens, so all baselines will be run by us. We will run RegionFocus with its published trigger rule and report its latency alongside accuracy. Our method beats it if it matches or exceeds RegionFocus accuracy on the real-clutter split at single-pass cost, and clearly exceeds it on occluded targets, where zooming cannot help.
The key ablation is random-overlay training. It uses the same four distractor families, the same number of distractor-augmented samples, the same reinforcement-learning recipe, the same general-data mixture, and the same number of tuning runs, but distractors are placed uniformly at random rather than mined. Mining wins only if it beats this ablation on the real-clutter split by more than the seed-to-seed variance across three seeds. Two further ablations isolate the remaining ingredients: removing the dismiss action, and replacing reinforcement learning with supervised training on the same mined screens.
Why not just…?
Why not just paste random pop-ups and look-alikes onto screenshots? This is the real threat. Data augmentation is cheap and might be enough. The only evidence against it comes from a different setting: RoTS's finding that random error injection underperforms policy-induced errors. Our random-overlay ablation settles the question directly. If it matches mining on real clutter, we report that and recommend it.
Why not just zoom at test time? RegionFocus triggers in 61.7% of trajectories with 66.8% latency overhead per trajectory, by its own account, and it cannot locate targets hidden behind modals. The occluded subset of our benchmark tests exactly that case.
Why not just read the accessibility tree and skip pixels? OSWorld reports that these trees reach millions of tokens on complex apps and that even GPT-4 with trees reaches only 12.24% success. Clutter such as notifications also appears inside the tree. We will include a tree-filtered agent on the platforms that expose trees, as a reference.
Why not just prompt the model to abstain when unsure, as DAS does for text? This is cheap and worth running. We will give the base policy an abstention instruction and an "unanswerable" action. We expect it to reduce false clicks without teaching it to dismiss occluders, but that expectation is untested.
Proposed Experiments
The decisive experiment trains ScaleCUA-7B under four conditions: no augmentation, random overlay, mined distractors, and mined distractors without the dismiss action. All conditions use desktop and web screens only. We then evaluate on the real-clutter split for every platform.
We measure four things:
- grounding accuracy;
- distractor capture rate, the fraction of errors landing on a distractor;
- over-dismissal rate on clean screens;
- retention on clean ScreenSpot-V2 and ScreenSpot-Pro, plus general multimodal checks.
Held-out mobile performance tests the transfer prediction. A reverse run that trains on Android and tests on desktop tests the asymmetry.
The end-to-end check runs the best checkpoint on OSWorld and AndroidWorld with notifications and background windows enabled versus disabled. For scale, RoTS-32B reports 47.4% on clean OSWorld Ubuntu, and ScaleCUA reports 60.6% on OSWorld-G.
We will confirm the main contrast at 32B scale. We will also sweep the distractor fraction from 5% to 40% to locate any over-dismissal ceiling analogous to the over-reflection ceiling RoTS found.
The proposal fails cleanly in two ways. If clean-to-clutter gaps are small, the parity concern is unfounded. If random overlays match mining, the policy-specific hypothesis is wrong.
References
- Recovering Policy-Induced Errors: Benchmarking and Trajectory Synthesis for Robust GUI Agents
- ScaleCUA: Scaling Open-Source Computer Use Agents with Cross-Platform Data
- OS-ATLAS: Foundation Action Model for Generalist GUI Agents
- GUI-Shift: Enhancing VLM-Based GUI Agents through Self-supervised Reinforcement Learning
- OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments
- Improving Zero-shot Reader by Reducing Distractions from Irrelevant Documents in Open-Domain Question Answering
- CompAct: Compressing Retrieved Documents Actively for Question Answering
- Visual Test-time Scaling for GUI Agent Grounding
- Intuitive Access to Smartphone Settings Using Relevance Model Trained by Contrastive Learning
- ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Use