Trained on What Screen Readers Cannot Read: Accessibility Debt in the GUI Corpora Behind Computer-Use Agents

Obada Kraishan, Kulsawasd JitkajornwanicharXiv 2026 · arXiv

TL;DR

AI agents designed to operate mobile apps are only as good as the "accessibility trees" they are trained on. This paper quantifies "accessibility debt"—the gap between what's on a screen and what developers actually label—finding that while newer datasets have improved, agents remain blind to nearly half of the interactive elements they need to use, with human-created benchmarks further masking the problem by favoring well-labeled targets.

What Problem Does This Paper Address?

To build an AI agent that can "use a phone," researchers typically train models on the accessibility tree: a structured record of every UI element (buttons, text fields, images) provided by the operating system. This is the same data stream used by screen readers like TalkBack.

However, developers are notoriously inconsistent at populating these trees. If an icon-based button lacks a content-description, a screen reader user hears nothing, and a computer-use agent "sees" an anonymous, unlabelled box. The authors call this shared failure accessibility debt.

The paper investigates how deep this debt runs in the massive GUI corpora used to train state-of-the-art agents. If our training data is missing labels for critical controls, agents inherit a massive blind spot. Furthermore, if the benchmarks we use to measure success are biased toward elements that do have labels, we might be overestimating how well these agents actually work in the wild.

Methods

The authors developed a rigorous automated coding scheme to analyze over 17 million interface elements. They defined a decision procedure to determine if an element "needs" a label (e.g., it's a tappable graphic with no text) and whether it actually "has" one.

Fig. 1. The label-status decision. Questions 2 and 3 route exempt elements past the debt test to the final split. The shaded outcome, unlabelled, is the accessibility debt.

Key methodological highlights include:

  • Role Classification: Normalizing class names to handle various Android frameworks (Material, AppCompat, etc.).
  • Hit-Testing: For datasets like AndroidControl that only record click coordinates, the authors implemented a point-in-bounds algorithm to resolve which specific UI element a human demonstrator actually touched.
  • Reliability: A two-coder manual check achieved high agreement ($\kappa \geq 0.91$), validating the automated pipeline.
  • Sensitivity Analysis: The authors re-ran their analysis under five different rule variations (e.g., discounting "placeholder" labels like "button123") to ensure their findings weren't artifacts of specific definitions.

Datasets

The study compares two landmark Android corpora collected seven years apart:

  1. Rico (2017): A broad crawl of over 9,300 free apps, contributing 66,261 screens.
  2. AndroidControl (2024): A more recent dataset focused on task demonstrations (e.g., "book a flight"), covering 700 app packages and 49,822 screen observations.

Key Insights and Ideas

  • The Benchmark Selection Effect: This is perhaps the most critical "aha" moment. Human demonstrators creating training data subconsciously favor elements they can identify. On screens containing multiple icon-like buttons, humans are 9 percentage points more likely to click one that is properly labeled. This means benchmarks are effectively "grading on a curve"—testing agents on the easiest-to-identify parts of the UI.
  • The Input Irony: Input controls (text fields, checkboxes, switches) are the least labeled category in both datasets. These are the very elements where the cost of a mistake is highest, yet they are the most likely to be "silent" in the accessibility tree.
  • Quality vs. Quantity: Older datasets like Rico didn't just have fewer labels; they had lower-quality ones. Descriptions in Rico were five times more likely to be useless placeholders (like "image" or "null") compared to AndroidControl.
  • Debt as Data Quality: By framing missing labels as "accessibility debt," the authors argue that this isn't just a developer problem—it's a data-quality problem for the AI research community.

Key Results

The quantitative gap is stark. Across the 17.3 million elements parsed:

  • Overall Coverage: Only 17.74% of elements needing a label had one in Rico, compared to 57.27% in AndroidControl.
  • Icon-like Elements: Even in the stricter "icon-only" category, coverage only reaches 62.75% in the modern dataset.
  • Input Controls: In Rico, a staggering 97.8% of input controls were unlabelled. In AndroidControl, this improved but still left 72.79% unlabelled.
Fig. 2. Label coverage across the two corpora. (a) Overall coverage among elements requiring a description. (b) Coverage by element role; exempt roles are omitted. (c) Within AndroidControl, elements touched by a task demonstration against the background population on the same screens. Hatching duplicates the colour encoding so the figure reads in greyscale.

Limitations and Future Work

  • Lower Bound on Coverage: The automated tree analysis cannot distinguish between a "meaningful" image (which needs a label) and a "decorative" one (which doesn't). Therefore, the coverage figures are likely conservative lower bounds.
  • Dataset Confounding: Because Rico was a broad crawl and AndroidControl was a curated set of task-oriented apps, the improvement in labeling might reflect better app selection rather than a general industry-wide improvement in accessibility over time.
  • Future Directions: The authors suggest that agent builders should treat pixel-based "screen recognition" (recovering semantics from images) as a necessary complement to the accessibility tree, rather than just an alternative. They also call for benchmark designers to stratify success rates by whether the target element was labeled, providing a more honest view of agent performance.

Summary updated

Abstract

Computer-use agents learn to operate mobile interfaces from corpora of captured screens and accessibility trees. The same trees are what a screen reader consumes, so an agent trained on them inherits whatever labelling gaps the underlying applications carry. This paper measures one of those gaps, content-description coverage, in two Android corpora: Rico, collected in 2017, and AndroidControl, collected in 2024. Across 17,373,614 interface elements, 17.74% of the Rico elements that need a content description under our coding rule carry one, against 57.27% in AndroidControl. The difference of 39.5 percentage points holds under five alternative coding rules and when whole apps are resampled. It describes two datasets that also differ in app selection, capture method and platform version; it does not measure change over time. Input controls are the least labelled role in both corpora (2.20% and 27.21%, or 30.99% in AndroidControl if hint text counts as a name). Descriptions in the older corpus are also five times more likely to be placeholders (10.25% against 1.95%), so the difference is not an artefact of developers typing filler. Resolving 26,205 task actions to the elements they touched shows that, among tappable graphics with no text, demonstrations operate elements described 9.0 points more often than the others on the screens they visit, about 3.5 points of it from the choice of element within a screen. That is a selection effect inside the benchmark. A two-coder reliability check (kappa = 0.91 and 0.96) and the full pipeline accompany the paper.
BibTeX
@misc{kraishan2026trained,
  author = {Obada Kraishan and Kulsawasd Jitkajornwanich},
  title = {{Trained on What Screen Readers Cannot Read: Accessibility Debt in the GUI Corpora Behind Computer-Use Agents}},
  year = {2026},
  eprint = {2611.16107},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.16107},
}
Figure