For an AI agent to operate a computer, it must bridge two fundamentally different worlds: the visual world of pixels that humans see, and the structural world of code that the operating system manages. Early digital agents often chose one or the other, relying either on text-based HTML/metadata or raw computer vision. However, recent research has converged on a more robust middle ground: the hybrid observation space.
This approach fuses raw visual screenshots with structured semantic metadata—such as accessibility trees or Document Object Models (DOM)—to create a representation that is more than the sum of its parts.
The Perception Gap in Digital Environments
The primary challenge in Graphical User Interface (GUI) automation is "grounding"—the ability to map a high-level intent (e.g., "Book a flight") to a specific set of coordinates and actions on a screen.
As Deanna D. Nguyen and Jian Chen note in their survey of GUI Agents, single-modality agents face a "complexity wall." Text-only agents are blind to visual cues like icons, layout-based meaning, or dynamic changes that aren't reflected in the code. Conversely, vision-only agents often struggle with precision; they might "see" a button but lack the semantic context to know its function if it isn't explicitly labeled with text.
Dual-Stream Perception: Pixels and Metadata
To solve this, researchers like Jing Yu Koh in VisualWebArena and Lingpeng Kong in OSWorld utilize a hybrid observation space. This architecture typically processes two parallel streams:
- The Visual Stream: A raw screenshot of the interface. This provides the spatial context, allowing the model to understand the layout and recognize non-textual elements like a "trash can" icon or a color-coded status bar.
- The Structural Stream: A pruned version of the accessibility tree (for OS/Mobile) or the DOM (for Web). This provides a hierarchical inventory of every interactive element, including its role (button, checkbox), state (checked, disabled), and unique ID.
By interleaving these modalities, models like WebGUM, developed by Hiroki Furuta and colleagues, can "read" the HTML while "looking" at the rendered page. This allows the agent to resolve ambiguities; if the HTML is poorly formatted or missing labels, the visual stream provides the necessary context to recover.
Grounding via Set-of-Mark Prompting
A key mechanism for making this hybrid data actionable is Set-of-Mark (SoM) prompting. In this technique, the agent doesn't just look at a raw image; the system overlays numeric tags or bounding boxes directly onto the screenshot, corresponding to elements found in the structural metadata.
In the OSCAR framework, Xiaoqiang Wang demonstrates how this "dual-grounding" bridges the gap between pixel-level layout and functional identity. When the agent decides to click "Button 5," it is using the semantic metadata to identify the button's purpose and the visual markers to ensure the click lands on the correct spatial coordinates. This method significantly reduces the "execution gap" where an agent knows what to do but misses the target.
Robustness through Redundancy
The intuition behind the hybrid approach is redundancy. Digital interfaces are notoriously "noisy." Accessibility trees can be incomplete, and screenshots can be cluttered.
Chaoyun Zhang and the team behind UFO found that integrating native OS APIs with vision-based perception allows agents to recover from errors that would fail a single-modality system. If a visual identification fails due to a custom-rendered UI, the agent can fall back on the underlying system identifiers.
Similarly, Christopher Rawles and Wei Li show in AndroidWorld that this hybrid representation is essential for navigating the high-dimensional decision space of mobile apps, where screen real estate is limited and UI elements are often densely packed.
Why It Works: Semantic-Spatial Alignment
The hybrid observation space works because it aligns semantic intent with spatial reality.
- The Structural Metadata tells the agent what is on the screen (the "what").
- The Visual Screenshot tells the agent where things are and how they relate to each other (the "where").
When these two are fused—whether through a transformer-based fusion module like in META-GUI or through interleaved text-and-image prompting—the agent gains a human-like understanding of the interface. It no longer sees a computer as a series of coordinates or a wall of code, but as a functional environment where actions have predictable, visual impacts.