Vision-language models that operate phone and computer interfaces fail in a specific, repeatable way. They say a button is on the screen when it is not, they predict that a tap will open a page it does not open, and they assume an action worked when the screen did not change. The usual remedy is synthetic training data. A large model explores an app, describes what it sees, and a smaller model is trained on those descriptions. This remedy has two problems. It is expensive, because the large model annotates every step. And it reproduces the error it is meant to fix, because the annotator is itself a vision-language model that hallucinates. We propose a different source of labels: the app's own behavior. A cheap crawler with no language model records which screen each executed action actually led to. The result is a directed graph whose nodes are screens and whose edges are actions that were really executed, and every fact in it is true because it was observed.
The proposal is to mine this graph for yes/no and multiple-choice probe questions and to use them as training data. Examples are "is there a control that opens Settings on this screen?", "which of these screens results from tapping here?", and "did this tap change anything?". The wrong answers are drawn deliberately from the graph's neighborhood: elements from the next screen over, the sibling screen reachable by a different button, or the same button on a different page. A language model is used only to turn verified structured facts into varied wording, never to decide what is true. The claim is that this graph-derived contrast is what teaches a small agent to stop hallucinating interface elements and action outcomes, and that it does so at a small fraction of the per-example cost of model-annotated data. The proposal also produces a GUI analogue of object-hallucination benchmarks, which does not exist today.
Core Claim
Three groups of prior work set up the gap.
The first motivates it. POPE turns object hallucination into binary probing with random, popular, and adversarial (co-occurrence) negatives, and finds a "yes" bias near 99% in some models. MERLIM inpaints objects away and finds models still assert them, with roughly an 8% precision drop. Both study natural photographs. Neither touches widgets, icons, or what an action does. GUI-PRA names "state-change blindness" in GUI judges. SeeClick shows that icons are much harder to ground than text.
The second group uses graphs to avoid hallucination at inference time. Executable Agentic Memory restricts an agent to edges of a prebuilt knowledge graph, so it cannot click widgets that do not exist. GUI-explorer retrieves observation–action–outcome triples at runtime. Both need a per-app graph at deployment, costing about 80 minutes of exploration for the first and about 28.5 seconds of ranking overhead per step for the second. Neither makes the model itself less prone to hallucinate on an app it has never seen.
The third group uses graphs to make training data. GUI-Bee builds an exploration graph with GPT-4o and has a multimodal model annotate transitions as instructions. Here the labels inherit the annotator's hallucinations, and the annotations are paid for per step. UI-Oceanus trains on forward dynamics, meaning it predicts the next screen from the current screen and action, over more than 20.8M transitions. That signal is execution-grounded, but it comes from a heavy multi-stage filtering pipeline and contains no explicit contrast against plausible but wrong alternatives.
Our proposal differs from all three. Labels come only from execution. Training is discriminative. The negatives are chosen by graph topology to be maximally confusable. The method is borrowed in spirit from Adaptive Problem Generation via Symbolic Representations, which edits math problems in a solver-checkable symbolic space and only then renders them into text. That work reports much larger structural diversity (embedding distance 0.37 vs 0.07) than natural-language edits. The UI graph plays the role of the symbolic space: every probe is generated and checked there, and an LLM only phrases it.
Research Program
Crawling. We crawl a set of Android apps that is disjoint from the evaluation apps, using an LLM-free crawler: random walk plus depth-first search with deduplication of near-identical screens. Deduplication uses perceptual hashing, as in GUI-explorer, and pixel and element-difference filtering, as in MobileVLM. Each edge stores the pre-screen, the action and its target element box from the accessibility tree, the post-screen, and whether the screen changed.
Probe families. From this graph we generate four kinds of probe.
Existence probes ask whether an element is present. Negatives come in three tiers. The easy tier draws elements from random other apps. The medium tier draws elements frequent in the same app, mirroring POPE's "popular" split. The hard tier draws elements present on graph-adjacent screens but absent from this one.
Outcome probes give a screen and an action and ask which screen follows. Distractors are siblings reachable from the same screen through a different edge, and the destination of a similar-looking element elsewhere in the app.
No-op probes use self-loop edges where an executed action left the screen unchanged. These target state-change blindness directly.
Erasure probes take the MERLIM idea and make it nearly free for interfaces. The view hierarchy gives each widget's exact bounding box, so we can fill that box with background and flip the existence label.
Naming icons by what they do. Icons without text labels are where grounding fails most. The graph lets us name them by function, for example "the control that opens the screen titled Recordings", using the destination node's title or activity.
Verbalization. A small open model turns each structured probe into several natural phrasings. A cheap check rejects any phrasing that mentions an element, screen, or attribute absent from the input record. This keeps the "verification tax" that DISCO and InstructDET pay with learned filters down to a string-level check.
Training. We fine-tune Qwen2.5-VL-7B on the probes mixed with standard agent data. The choice of student follows the reported baselines in the file.
Benchmark. A held-out split of probes from unseen apps, with a 500-item human audit to measure residual label noise, becomes a released GUI hallucination benchmark.
Accounting. Throughout, we report cost per retained training example in emulator-seconds plus generation tokens, and the fraction of generated items discarded. Overhead is a stated axis of the claim, not an afterthought.
The main known risk is wrong state abstraction. LLM-Explorer reports functionally distinct elements merged into one abstract state. A bad merge turns a true negative into a false one. The human audit measures this directly. We also vary the deduplication threshold to see how much label noise the method tolerates.
Baselines
Naive reference. The do-nothing reference is the unmodified Qwen2.5-VL-7B. The file reports 38.04% success for it on MobileMiniWoB++ (GUI-PRA). We will measure it ourselves on AndroidWorld, ScreenSpot, GUI-KRB, and the new probe benchmark.
Model-annotated transition data in the style of GUI-Bee. This is the closest competitor in data recipe. We will run GUI-Bee's annotation step, an MLLM describing each recorded transition as instruction data, on the same crawled graph. The two arms will have the same number of training examples and the same fine-tuning budget, and both start from the same base model. This isolates where the labels come from. We will also report the annotation cost of this arm. The file gives no specific NovelScreenSpot numbers for GUI-Bee ("no reported result found on Lacuna" beyond relative statements), so this comparison rests on our own reruns. Beating it means lower hallucination on the probe benchmark and equal or better ScreenSpot grounding at lower cost per example.
Forward-dynamics training in the style of UI-Oceanus. This is the strongest competitor in signal type, because its labels are also execution-grounded. We train the same base model to predict the post-screen's structured description from the pre-screen and action, on exactly the same transitions. Compute is matched in GPU-hours. UI-Oceanus reports 56.9% on unseen mini-programs and a 16.8% online navigation gain. Those figures are from a different ecosystem and are not directly comparable, which is why we rerun on matched data.
Strongest reported agent numbers on AndroidWorld. These are reference points rather than same-budget rivals. GUI-Reflection reaches 34.5% with InternVL2.5-8B (GUI-Reflection). GUI-explorer reaches 47.4% with runtime retrieval (GUI-explorer). Darwinian Memory reports 66.4% using a 72B model and persistent memory (Darwinian Memory). For grounding, SeeClick reports 53.4% on ScreenSpot. Since our contribution is a training signal and not an agent architecture, the fair claim is about gains over the same base model with the same agent loop. Beating the reported numbers is a stretch outcome, not the bar.
Ablations. Two ablations isolate the key ingredient. In the first, we keep everything fixed but draw all negatives at random rather than from graph neighbors. In the second, we drop the verbalizer and use fixed templates. If topology-mined negatives matter, the first ablation should lose most of the gain on the hard probe tier and on GUI-KRB.
Why not just…?
Why not just let a frontier model annotate the transitions, as GUI-Bee does? That approach puts a hallucinating annotator in charge of labels. MERLIM and POPE give direct evidence that such annotators assert absent objects. LVLM-as-judge studies (Chen et al.) report hallucinated visual defects in automated judging. GUI-Bee itself notes high API latency and costs that grow with context. The matched-count comparison above settles whether execution labels are better and not merely cheaper. The human audit gives label error rates for both arms.
Why not just train on forward dynamics, which is also verified by execution? This is the real threat. UI-Oceanus shows forward dynamics beats inverse dynamics and scales log-linearly. Forward dynamics might teach outcome knowledge as well as our outcome probes do. Our bet is narrower. Next-state prediction never forces the model to reject a near-miss: the sibling screen, the neighbor's element, the tap that did nothing. We expect the hallucination gap to show up on the adversarial tier and the no-op probes. The compute-matched comparison decides it. If forward dynamics matches us on the hard tier, the method reduces to a cheap benchmark plus a data-efficiency result, and we will report it that way. A combined arm tests whether the two are complementary.
Why not just constrain the agent to a graph at inference, as Executable Agentic Memory does? Graph-constrained search removes invalid actions only on apps that have already been crawled, and costs a cold start per app plus a graph that goes stale when the app updates. Our evaluation on apps never crawled, NovelScreenSpot web environments and held-out Android apps, measures exactly what inference-time constraints cannot provide.
Why not just inject errors into successful trajectories, as GUI-Reflection does? Error injection trains recovery after a mistake. Our probes train perception before the mistake. The two are compatible. We will report GUI-Reflection-style data with and without our probes added, to check that the effects stack rather than overlap.
Proposed Experiments
The first decisive check is the GUI hallucination benchmark itself. Before any training, we measure base-model "yes" rates and F1 per negative tier. If GUI models show the POPE pattern, with accuracy dropping from random to adversarial negatives, the probe design is validated. If they do not, the premise is weaker than claimed.
The second check trains the main model and every baseline arm on the same crawl. We report probe-benchmark F1 per tier, ScreenSpot accuracy (icon versus text separately, given SeeClick's finding), GUI-KRB error, and success on AndroidWorld and MobileMiniWoB++ under one fixed agent loop. Each arm's cost per retained example sits alongside its accuracy.
The third check is the negative-sampling ablation together with the abstraction-threshold sweep. These test whether the gains come from graph topology and how much state-merging noise the method can absorb.
The proposal succeeds if three things hold together. Execution-labeled probes beat model-annotated data at equal count. They beat forward dynamics on the adversarial and no-op tiers at equal compute. And they do so at a per-example cost dominated by emulator time rather than model tokens.
References
- Evaluating Object Hallucination in Large Vision-Language Models
- Behind the Magic, MERLIM: Multi-modal Evaluation Benchmark for Large Image-Language Models
- GUI-PRA: Process Reward Agent for GUI Tasks
- SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents
- Executable Agentic Memory for GUI Agent
- GUI-explorer: Autonomous Exploration and Mining of Transition-aware Knowledge for GUI Agent
- GUI-Bee: Align GUI Action Grounding to Novel Environments via Autonomous Exploration
- UI-OCEANUS: Scaling GUI Agents with Synthetic Environmental Dynamics
- Adaptive Problem Generation via Symbolic Representations
- MobileVLM: A Vision-Language Model for Better Intra- and Inter-UI Understanding
- DISCO: Distilling Counterfactuals with Large Language Models
- InstructDET: Diversifying Referring Object Detection with Generalized Instructions
- LLM-Explorer: Towards Efficient and Affordable LLM-based Exploration for Mobile Apps
- GUI-Reflection: Empowering Multimodal GUI Models with Self-Reflection Behavior
- Darwinian Memory: A Training-Free Self-Regulating Memory System for GUI Agent Evolution
- MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmark