The GUI Is Not the State: Diagnosing State Aliasing in GUI World Models

Dongsheng Liu, Chao Jin, Wenkui Yang, Hejin Wang, Junwei Yang, Zeren Zhang, Ziwei Chen, Huaibo Huang, Jie Cao, Ran HearXiv 2026 · arXiv

TL;DR

This paper identifies state aliasing—a fundamental flaw where GUI World Models fail because they only consider the current screen, ignoring hidden information like clipboard contents or background file changes. The authors introduce StateAliasBench to diagnose this issue and propose a lightweight "predictive-state recovery" mechanism that allows existing, frozen world models to account for hidden history, significantly improving their predictive accuracy and the performance of downstream autonomous agents.

What Problem Does This Paper Address?

Imagine an AI agent trying to predict what happens when it clicks "Paste" in a text editor. If the agent only looks at the current blank screen, it has no way of knowing what text will appear. The "state" of the environment (the clipboard content) is hidden. This is state aliasing: two different histories lead to the exact same visual observation, but require different future predictions.

Most current GUI World Models (GUI-WMs) are "observation-only," meaning they predict the next screen based solely on the current screenshot and action. The authors argue that the GUI is not the state. By ignoring transition-relevant hidden information—like whether a file was successfully moved or if a timer is about to expire—these models make systematic errors that hinder agent planning and simulation.

Figure 1: State aliasing and predictive-state augmentation. Observation-only conditioning can alias histories with different transition-relevant states (left). We recover predictive state from history and use it to condition a frozen GUI world model for state-consistent prediction (right).

Methods

The authors tackle this problem with a three-part framework:

  1. Diagnosis (StateAliasBench): They created a benchmark using "strict pairs." A strict pair consists of two interaction branches that share the same current observation and next action but have different hidden states and different ground-truth outcomes.
  2. Predictive-State Recovery: Instead of retraining massive world models, the authors developed a lightweight Unified Predictive-State Estimator ($G_\phi$). This model looks at the interaction history and recovers a structured JSON representation of the hidden state (e.g., {"clipboard_text": "Hello World"}).
  3. Deterministic State Interface: To make this recovered state usable by frozen existing world models, they designed a deterministic interface. This interface translates the recovered state into a model-compatible prompt (like a "Files blueprint" or a specific "notification-shade" description) that guides the frozen GUI-WM to generate the correct next state.

To train the estimator, they used Multi-Teacher Distillation (MTD), where specialized "teacher" models for different state types (like timers vs. clipboards) pass their knowledge to a single, unified student model.

Figure 2: The framework involves constructing strict pairs from real traces, training a unified estimator via distillation, and using a deterministic interface to augment frozen world models.

Datasets

  • StateAliasBench (New): A diagnostic benchmark featuring 800 verified strict pairs across 25 causal templates. It focuses on four "state families":
  • Persistent Artifact: File system changes (existence/location).
  • Clipboard: Text payloads stored in memory.
  • Ordered Collection: The specific order of items in a playlist or queue.
  • Temporal Deadline: Timer states (running, fired, or paused).
  • AndroidWorld: An existing general-purpose GUI agent benchmark used to test if these improvements translate to better performance in complex, multi-step tasks.
Table 1: State families in StateAliasBench. ⊥denotes absence or an empty state.

Key Insights and Ideas

  • The "Observation-Only" Failure: Existing GUI-WMs (like Code2World and gWorld) achieve 0% accuracy on the primary diagnostic metric (PairCorrect) because they cannot distinguish between aliased states.
  • History $\neq$ State: Simply feeding the raw interaction history into a world model isn't enough. The models often fail to attend to the right historical cues. Explicitly recovering structured state is a much more effective way to bridge the gap.
  • Augmentation over Retraining: You don't need to retrain a 7B or 8B parameter world model to fix state aliasing. A small, specialized estimator (like the 2B Qwen-based one used here) can "patch" the frozen model via the prompt interface.
  • Cross-App Challenges: The hardest form of distribution shift for these models is moving between different applications, which changes the visual context even if the underlying state logic (like a clipboard) remains the same.

Key Results

  • Restoring Predictive Power: On StateAliasBench, the proposed method raised the PairCorrect score from 0% (observation-only) to approximately 80% across different state families.
  • Generative Fidelity: Augmenting the models didn't just fix the logic; it also improved the visual quality of the predictions. Metrics like DINOv2 and SigLIP similarity showed significant gains (e.g., +0.15 DINOv2 for Code2World).
  • Downstream Agent Success: When used to assist agents in AndroidWorld, predictive-state conditioning provided a Relative Gain of 66-75% over the improvement gained by adding a basic world model alone.
Table 2: Quantitative results showing that while standard models fail completely on aliased pairs, the Predictive State method restores high accuracy.

Limitations and Future Work

  • Scope of State: The study focuses on four specific state families. Real-world GUIs have many more hidden variables (e.g., network latency, background sync, user permissions) that aren't yet covered.
  • Residual Error: Even with state recovery, the frozen world models still have their own inherent generation errors. The authors note that providing the state reduces ambiguity but doesn't perfectly recover the underlying "transition law."
  • Future Direction: The authors suggest that future GUI world modeling should move toward architectures that natively account for hidden transition states rather than relying on post-hoc patches.

Summary updated

Abstract

GUI World Models (GUI-WMs) are increasingly used to predict future states for agent planning and simulation, yet most existing formulations condition only on the current GUI observation and action. We identify state aliasing, where the vis- ible interface omits transition-relevant environment state, so identical observable conditions can correspond to different valid futures. To diagnose this failure mode, we introduce StateAliasBench, a diagnostic benchmark that explicitly isolates such ambiguities via strict pairing. We further propose lightweight predictive- state recovery that infers structured state from history and augments otherwise frozen GUI-WMs through a deterministic state interface. Family-specific special- ists provide state recovery across heterogeneous state types, and multi-teacher dis- tillation consolidates them into a single unified estimator. Experiments show that existing GUI-WMs exhibit systematic failures under observation-only condition- ing, while predictive-state augmentation substantially restores state-sensitive pre- diction across evaluated WMs, preserves generative fidelity, and improves down- stream performance of GUI agents on AndroidWorld. These results suggest that reliable GUI world modeling should account not only for what is visible, but also for the hidden transition state that determines what happens next.
BibTeX
@misc{liu2026gui,
  author = {Dongsheng Liu and Chao Jin and Wenkui Yang and Hejin Wang and Junwei Yang and Zeren Zhang and Ziwei Chen and Huaibo Huang and Jie Cao and Ran He},
  title = {{The GUI Is Not the State: Diagnosing State Aliasing in GUI World Models}},
  year = {2026},
  eprint = {2609.32679},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2609.32679},
}
Figure