Structured Textual Representations for GUI Automation

25 concepts19 papers1 proposal

When an autonomous agent interacts with a graphical user interface (GUI)—whether a mobile app, a webpage, or a desktop application—it faces a fundamental representation problem. Traditionally, developers have approached this problem from two extremes: raw visual pixels or raw underlying source code.

Both extremes present severe bottlenecks for modern Large Language Models (LLMs) and Multimodal LLMs (MLLMs). On one hand, passing raw screenshots to visual encoders introduces resolution bottlenecks; standard downsampling often turns fine-grained details, like menu items or small text labels, into unreadable pixel mush. On the other hand, passing raw source code (such as deep Document Object Model (DOM) trees or verbose Android XML layouts) floods the model's context window with non-essential styling information and redundant container nodes, obscuring the functional intent of the interface.

To solve this, a prominent research direction has emerged: structured textual representations. By transforming complex visual layouts and hierarchical code into clean, semantic, and compact text abstractions, researchers are enabling language models to effectively parse, reason about, and automate user interfaces.


The Core Mechanism: Mapping to Web Syntax

The primary reason structured textual representations work so well lies in the nature of LLM pre-training. Because language models are trained on vast web corpora containing billions of web pages, they possess an innate understanding of HTML and web structures.

To exploit this pre-existing knowledge, researchers frequently translate non-web interfaces—such as native mobile applications—into simplified HTML or XML syntax. As demonstrated by Bryan Wang et al. in their work on Enabling Conversational Interaction with Mobile UI using Large Language Models, mapping native mobile components (like TextView or Button) to standard web tags (like <p> or <button>) allows models to generalize their web-derived reasoning to mobile tasks without requiring massive, task-specific datasets.

Native Android Hierarchy          Simplified HTML Representation
┌──────────────────────────┐      ──────────────────────────────
│ LinearLayout             │      <form>
│   TextView ("Username")  │ ───>   <p>Username</p>
│   EditText               │        <input type="text" />
│   Button ("Submit")      │        <button>Submit</button>
└──────────────────────────┘      </form>

This structural mapping strategy is widely adopted across the literature:

  • Guohong Liu et al. developed AutoDroid, which converts Android UIs into simplified HTML-style text to help LLMs process complex interface structures and generate accurate actions.
  • Shihan Deng et al. utilized a similar transformation in Mobile-Bench, demonstrating that substituting raw XML with simplified HTML significantly improves an agent's structural understanding and element identification during autonomous navigation.
  • For multi-user testing environments, Sidong Feng et al. implemented a pruned, simplified XML hierarchy in Agent for User to strip away non-essential attributes and boost LLM comprehension speeds.

Filtering Noise via Accessibility Trees

Even when dealing with native web applications, raw source code is often too bloated for efficient processing. To extract a cleaner semantic representation, researchers look to the accessibility tree rather than the raw DOM.

Accessibility trees are operating-system-level abstractions designed originally for screen readers. They inherently strip away purely decorative elements, layout containers, and redundant styling nodes, leaving behind a pure functional hierarchy of interactive elements.

In WebUI: A Dataset for Enhancing Visual UI Understanding with Web Semantics, Jason Wu et al. demonstrated that training models on accessibility tree data helps filter out visual noise and accurately isolates functional roles, such as identifying when a generic container is acting specifically as an interactive button or input field. Similarly, Junpeng Liu et al. leveraged accessibility trees in Harnessing Webpage UIs for Text-Rich Visual Understanding to build their MultiUI dataset, proving that this refined abstraction creates a highly efficient mapping between visual components and their functional roles.

By flattening these hierarchical accessibility trees into structured text strings, agents gain a searchable, lightweight map of the interface. As highlighted by Noam Kahlon et al. in Agent-Initiated Interaction in Phone UI Automation, this text-based map allows agents to reliably identify interactive components and determine when to ask users for clarification without relying solely on computationally expensive visual processing.


Reconstructing Structure from Vision

What happens when the underlying code or accessibility tree is inaccessible, such as in closed-source applications, legacy desktop software, or static video streams? In these scenarios, researchers use computer vision pipelines to reconstruct the structured textual representation from raw pixels.

┌──────────────────┐     OCR & Icon     ┌──────────────────────────────────────┐
│  Raw Screenshot  │ ─────────────────> │ Structured Text Schema               │
│  [ Search  [Q] ] │     Detection      │ - Input Field: "Search" (x1, y1)     │
└──────────────────┘                    │ - Icon Button: "Search Lens" (x2, y2)│
                                        └──────────────────────────────────────┘

This hybrid approach combines the perceptual strengths of computer vision with the reasoning strengths of text-based language models:

  • OCR and Icon Detection: In Dynamic Planning for LLM-based Graphical User Interface Automation, Shaoqing Zhang et al. transformed raw visual inputs into HTML-like screen captions using optical character recognition (OCR) and icon detection, allowing text-only models to effectively reason over complex layouts.
  • Multimodal Fusion: Raghav Kapoor et al. introduced OmniACT, which converts screenshots into text by combining OCR for text, template matching for icons, and hue-based categorization for color-coded elements.
  • Deceptive Pattern Detection: In Automatically Detecting Online Deceptive Patterns, Asmit Nayak et al. combined extracted text and detected UI elements into a structured, text-only representation to feed into downstream language models for dark-pattern classification.
  • Screencast Analysis: For dynamic video data, Wentian Zhao et al. showed in Video Question Answering on Screencast Tutorials that representing UI elements as collections of OCR-derived text captures semantic differences far better than standard visual feature extractors, distinguishing between panels that look visually identical but serve entirely different software functions.

Even in early automated testing frameworks like Humanoid, Yuanchun Li et al. recognized the value of abstraction, transforming raw screens into simplified two-channel representations that separated text regions from non-text elements to eliminate visual noise like colors and font styles.


Predictive and Stateful Textual Schemas

Structured text abstractions are not limited to static snapshots; they are increasingly used to model dynamic interface changes and perform lookahead planning.

When an agent interacts with a GUI over time, processing a continuous stream of high-resolution video frames creates massive computational overhead. To bypass this, Stefano Petrangeli et al. introduced ScreenLLM, which encodes dynamic GUI changes into a compact, textual "stateful screen schema." By converting cursor coordinates, text changes, and keyframes into text, the schema preserves fine-grained details that would be lost during image downsampling while avoiding the scalability issues of processing long screen recordings.

Taking this a step further, text abstractions can serve as the foundation for generative "world models" where an agent simulates the future. In MobileDreamer: Generative Sketch World Model for GUI Agent, Yilin Cao et al. designed a framework where agents "imagine" the visual consequences of their actions before executing them. Instead of predicting future raw pixels—which is computationally expensive and prone to hallucination—the model generates future screen states as structured textual sketches containing element labels, text content, and bounding boxes.

Current State (Text) ──> Action ──> World Model ──> Predicted Next State (Text Sketch)
[Button: "Next"]         Click      (MobileDreamer)  [Label: "Success Page", Box: [0,0,100,50]]

Why the Idea Works: A Summary

Structured textual representations work because they enforce a clean separation of concerns. They strip away non-functional visual styling (colors, fonts, gradients) and boilerplate code layout structures, leaving behind a dense, high-signal representation of functional intent and spatial hierarchy.

By converting this distilled information into formats like simplified HTML, XML, or textual schemas, the field bridges the gap between visual user interfaces and text-based language models. As articulated by Kristian Kolthoff et al. in Interlinking User Stories and GUI Prototyping, these semantic text abstractions allow models to bypass raw code and pixel complexities, focusing their computational capacity entirely on high-level reasoning, planning, and action execution.

Key Foundational Work

Migratable User Interface Descriptions in Component-Based Development (2002) — contributed 1 concept elements to this direction.
Humanoid: A Deep Learning-based Approach to Automated Black-box Android App Testing (2019) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_semantic_ui_representation_4cf86b14,
  title = {semantic UI representation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/semantic-ui-representation/rdr_4cf86b14259846a2a5f97828c3f55ad9},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_semantic_ui_representation_4cf86b14_v1,
  title = {semantic UI representation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/semantic-ui-representation/rdr_4cf86b14259846a2a5f97828c3f55ad9/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake