Practice in Text, Act in Pixels: Correcting Where Perception Fails

Zhongling Xu, Hairong WangarXiv 2026 · arXiv

Training agents to navigate graphical user interfaces (GUIs) is a classic "sim-to-real" problem. While it's incredibly cheap to let an agent "practice" using structured text descriptions of a screen (like a list of buttons and their labels), real-world deployment requires the agent to "act" on raw pixels. This transition usually relies on a vision-language model acting as a screen reader.

The problem? Even the best low-cost screen readers are prone to "hallucinations" and omissions, leading to a massive performance drop when moving from text to pixels. This paper from researchers at UT Austin investigates how to bridge this "text-to-pixels gap" with a surprising conclusion: don't retrain the agent; fix its eyes.

TL;DR

GUI agents trained on structured text fail on screenshots because vision-language screen readers make systematic, recurring errors. This paper demonstrates that spending a small budget of labeled screenshots to align the reader (fine-tuning it to match the agent's expected text view) is significantly more effective than training the agent's policy to be robust to those errors, closing the performance gap by up to 85%.

What Problem Does This Paper Address?

In GUI automation, there is a massive cost disparity between training in "text" (using element lists from a simulator) versus "pixels" (rendering and processing screenshots). Training in text is roughly 7x faster and much cheaper. However, at test time, structured text (like HTML or accessibility trees) is often unavailable or messy. Agents must use a screen reader to interpret screenshots.

The authors identify a text-to-pixels gap: a policy that succeeds 87.5% of the time when given perfect text might only succeed 43.0% of the time when relying on a screen reader. The core challenge is determining the most data-efficient way to use a limited budget of real screenshots (e.g., 2,000 images) to close this gap.

Methods

The researchers compared several strategies for spending their "screenshot budget":

  1. Noise in Practice: Injecting random noise into the text-based training to simulate reader errors.
  2. Direct Screenshot Training: Fine-tuning the agent's policy (the "brain") directly on screenshots.
  3. Reader Alignment (The Proposed Solution): Keeping the policy fixed and fine-tuning the screen reader (the "eyes") using LoRA. The reader is trained to output the exact text view format and layout order that the policy practiced on.
Figure 1. Spending the same budget of about 2,000 screenshots on aligning the reader gives clearly higher success than spending it on the policy. Left: the budget can go to practice (a; Sec. 4.4), the policy (b; Sec. 6.1) or the reader (c; Sec. 5). Middle: an illustration of a missed element, a misread state and the reader’s own order (Fig. 2 shows real misreads). Right: full-task success on new instances (mean ± std, two training seeds); dark red: the policy adapted with (c)’s labels instead; dashed: the no-screenshot policy in text.

The setup used Qwen3-8B for the policy and Qwen3-VL-4B for the screen reader.

Datasets

  • MobileGym: A simulated platform for mobile tasks (e.g., "Turn off the 7:00 alarm") across apps like WeChat, Alipay, and Clock. This provided the "cheap" text views and ground-truth labels.
  • AndroidControl: A dataset of human demonstrations on real Android apps, used to validate if the findings hold up on real-world, non-simulated data.

Key Insights and Ideas

  • Errors are Systematic, Not Random: The most critical finding is that screen reader errors are "sticky." If a model misreads a specific icon once, it will likely misread it again in a different episode. Figure 2. Comparison showing that real reader errors (left) are systematic and tied to specific elements, whereas standard noise (right) is scattered.

  • Sim-to-Real Noise Fails: Because errors are systematic, traditional domain randomization (adding random noise to text) fails. It doesn't teach the policy which elements are likely to be misread, just that some elements might be wrong.

  • Layout Order is a Hidden Cost: Beyond just misreading text, screen readers often list elements in a different order than the standard top-to-bottom layout. This "reordering" alone accounts for a significant chunk of the performance drop.
  • Perception Alignment is Universal: A single aligned reader can improve any policy trained on that text format. You don't need to retrain your expensive agent every time you get a few more screenshots; you just refine the interface between the pixels and the agent.
Figure 2. The reader misreads largely the same elements across episodes; rate-matched noise mostly corrupts other elements. A schematic of one calibration screen (Sec. 3), drawn from its text view (bars: text; squares: icons; screenshot in supplemen- tary Fig. S2) and chosen by a rule fixed in advance (supplemen- tary Sec. F). Color: how often the original reader misreads each element in calibration episodes; never-misread elements are gray unless misread here. Colored fill: misread on this read or in this noise draw. The noise draw puts 19 of its 24 misreads on never- misread elements; the noise matches the reader’s error rates over the calibration screens, not per screen.

Key Results

  • Superior Success: With a budget of 2,000 screenshots, Reader Alignment achieved 80.9% full-task success, compared to only 55.6% for direct policy training.
  • Closing the Gap: Alignment closed 85% of the text-to-pixels gap for the primary "clean-text" policy.
  • Data Efficiency: Aligning the reader with just 500 screenshots (63.4% success) outperformed training the policy on 2,000 screenshots.
  • Error Reduction: On held-out screenshots, alignment slashed the element error rate from 25.6% down to just 1.6%.

Limitations and Future Work

  • Generalization to New Apps: While alignment works wonders for the apps and elements seen during fine-tuning, the gains are much smaller on entirely "held-out" apps. The reader needs to be aligned on the specific apps the agent will use.
  • Label Availability: The method relies on having ground-truth text views for the alignment screenshots. While simulators provide this for free, getting high-quality labels for real-world devices (where accessibility trees are often broken) remains a hurdle.
  • Real-Device Performance: On the real-world AndroidControl dataset, alignment improved the error rate but didn't reach the same near-perfect levels seen in the simulator, suggesting that real-world visual complexity and label noise are still significant challenges.

Summary updated

Abstract

GUI agents can learn cheaply from a text view of an interface and act on screenshots through a vision-language screen reader. Yet on a simulated mobile platform with a low-cost reader, policies that complete most tasks in text fail on many of the same tasks on screenshots. We ask where a small budget of real screenshots is best spent to close this gap. We find that the reader's misreads are systematic and recur on the same interface elements. Noise matched to the reader's error rates, a natural sim-to-real remedy, misses this structure, and in a pre-specified comparison it did not beat training the policy directly on screenshots. We therefore correct the misreads at their source with reader alignment, which fine-tunes the reader on screenshots paired with their text views. The reader learns to output each screen's text view, as in text practice, and the policy stays unchanged. Spent on the reader, the screenshot budget gives 80.9% full-task success on new task instances, against 55.6% when spent on training the policy. Given the same labeled screenshots, adapting the reader also beats adapting the policy, and a single aligned reader improves every text-trained policy we tried without retraining any of them.
BibTeX
@misc{xu2026practice,
  author = {Zhongling Xu and Hairong Wang},
  title = {{Practice in Text, Act in Pixels: Correcting Where Perception Fails}},
  year = {2026},
  eprint = {2611.22722},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.22722},
}
Figure