Screen-reading agents are AI systems that operate phones and computers by looking at screenshots and choosing where to tap or type, without reading any of the hidden code behind the interface. They are increasingly asked to do multi-step work: open a job applicant's profile, scroll through a reference letter, check a portfolio, then return to a form and click "advance" or "reject." The agent sees the evidence that should drive its decision early in the task. By decision time, that evidence sits somewhere in the middle of a long history of screenshots. We already know that language models use the middle of long inputs poorly. We also know that when they lack information, they fall back on societal stereotypes. This proposal tests whether these two facts combine in screen-reading agents. Our hypothesis is that when an agent loses track of individuating evidence because of where it sits in the history, it fills the gap with a stereotype cued by whatever demographic signal is on the current screen, such as an applicant's name or photo.
The claim is narrow, and it can be measured. We predict that the demographic gap in an agent's actions depends on position. It should be near zero when the decisive evidence is visible on the current screen, grow as the evidence moves into the middle of the screenshot history, and partly recover when the evidence is at the very start of the history. The gap should also track the agent's evidence-use accuracy on matched control tasks that contain no social content. If this holds, part of agent bias is a memory-failure problem, and the remedies come from long-context and temporal-grounding research rather than from alignment prompting alone. If it fails, so that bias is flat across positions, then bias in these agents is a fixed prior, and memory fixes will not touch it. Either outcome is informative.
Why this is open
Three bodies of work each come close to this question without asking it. The first concerns long GUI tasks. LongHorizonUI documents a "performance cliff," with success above 90% at 5 steps and below 60% past 15 steps. It treats the cliff as cascaded execution error and repairs it with reflection and state rollback. History-Aware Reasoning for GUI Agents compresses past steps into text summaries (Act2Sum) and trains with reinforcement learning. Neither paper asks whether the model's attention to old screenshots decays with position, and neither measures bias.
The second concerns position in long inputs. Lost in the Middle and Serial Position Effects of Large Language Models establish a U-shaped accuracy curve over input position. Positional Contrastive Decoding shows that the correct answer is often still among the model's top few candidates, but is outcompeted by distractor tokens that sit closer to the query. All of this work is text-only. Its own stated open problem is how the effect behaves over interleaved image-token blocks such as screenshot histories.
The third concerns bias under load. Uncovering Stereotypes in Large Language Models finds that stereotyping rises and refusals fall as task complexity grows. FairBelief finds that harmful beliefs sit at the peak of the model's predicted distribution, so they are the model's default completion. That is exactly what would surface once specific evidence is lost. These papers motivate the hypothesis, but they treat complexity as an undifferentiated knob. We isolate one specific, controllable component of complexity: how far back in the history the evidence lies.
The closest agent-bias work does not touch the mechanism either. AuditAgent uses GUI agents to audit platforms for bias, not the agents themselves. Can Large Language Model Agents Simulate Human Trust Behavior? finds demographic sensitivity in agent decisions, but in short text games with no history structure.
Core Claim
Fix the decision task. Let $p$ be the position of the decisive evidence screen in the agent's history, measured in steps before the decision screen. Every task instance comes in a counterfactual pair that is identical except for the demographic cue $d \in \{A, B\}$ shown on the decision screen. We define the positional bias gap as
$$G(p) = P(\text{favorable action} \mid d = A, p) - P(\text{favorable action} \mid d = B, p),$$
and the evidence-use rate $U(p)$ as decision accuracy at the same position on structurally identical tasks whose content is non-social, such as choosing between two products. The core prediction is that $G(p)$ is small where $U(p)$ is high and grows where $U(p)$ dips. In other words, $G$ and $U$ should be negatively correlated across positions within a model.
The sharper quantity is the stereotype fill-in rate. Take only the instances where the agent's action is inconsistent with the evidence, which we read as evidence lost. On those instances, measure whether errors lean toward the stereotyped direction. A neutral agent that loses evidence errs symmetrically across $d$. A biased agent errs in the stereotyped direction. This conditioning separates "forgot the evidence" from "forgot the evidence and substituted a prior." Generic accuracy loss cannot produce the second pattern.
Research Program
Benchmark construction. We build a controlled benchmark of multi-screen decision tasks in instrumented mock apps: an applicant tracker, a loan-review dashboard, and a rental-application inbox. The setup follows the state-inspection evaluation style of AndroidWorld, so success is read from app state rather than judged by a model. Each episode has three kinds of screens. One evidence screen carries the decisive information, for example a qualification that clearly meets or fails a stated requirement. A variable number of neutral navigation screens act as filler. The final decision screen shows the demographic cue and the action buttons.
The demographic cues are names rendered in the screenshot, with name lists that signal gender and ethnicity, plus optional avatar images. Following the "bias-bleaching" logic of A Unified Framework and Dataset for Assessing Societal Bias in Vision-Language Models, we also run a cue-free variant to establish each model's baseline action rate. We sweep the evidence position $p$ from 0 (evidence on the decision screen) up to about 25 steps back. This range brackets the 22.1-step average of LongGUIBench. Every social task has a matched non-social twin with identical layout and step structure.
History representations, where the temporal-boundary question enters. The same episode is presented to agents under four ways of carrying the past forward. The first is the full raw screenshot history. The second is a sliding window of the last $k$ screenshots, which creates a hard temporal boundary. The third is a text-summary history in the style of Act2Sum. The fourth is an accessibility-tree text history, meaning the machine-readable list of on-screen elements that operating systems expose, used here instead of pixels. The sliding window lets us study boundary sensitivity directly. Video work shows that a boundary placed one segment off can permanently discard the needed context (RGNet, Learning to Locate Visual Answer in Video Corpus Using Question). We test whether $G$ jumps discontinuously when the evidence screen crosses the window edge, as opposed to the smooth middle-decay of the full-history condition. The accessibility-tree condition addresses the contradiction raised in OSWorld, where text trees helped GPT-4 most but inflated token counts massively. Pixel and text histories may differ in where the middle-loss sets in, and therefore in where bias fills in.
Models. We evaluate open-weight screen agents, including Qwen2.5-VL-7B and the HAR-GUI-3B and UI-TARS family of checkpoints where weights are available, plus at least one commercial multimodal model through its API. Open weights let us apply logit-level interventions and inspect whether the evidence-consistent action is still among the top candidates when the model gets it wrong. That inspection is the screenshot analogue of the posterior-salience finding in Positional Contrastive Decoding.
Interventions as causal tests. If position is the cause, then fixing position should shrink the gap, and fixing it in different ways should shrink it in proportion to the recovery in $U(p)$. We test three interventions. The first is oracle re-presentation, which copies the evidence screenshot to just before the decision. It serves as an upper bound and a causal check. The second is learned re-retrieval, in which a lightweight scorer conditioned on the current screen selects past screenshots to re-attend to before acting. It borrows the joint retrieve-and-ground training idea from RGNet and Grounded Question-Answering in Long Egocentric Videos rather than a separate fixed retriever. The third is positional contrastive decoding, applied to open models whose position encoding is rotary-compatible, so that it works on action tokens.
Baselines
The naive reference point is the unmodified agent with its default history format, evaluated on our benchmark across all positions. It defines the $G(p)$ and $U(p)$ curves that everything else must change. There is no reported result found on Lacuna for any agent on this positional-bias benchmark, because it does not yet exist. Running this baseline is therefore itself a primary result.
The strongest existing method for long screenshot histories in an open model is HAR from History-Aware Reasoning for GUI Agents. It reports a step success rate of 62.31% on GUI-Odyssey for HAR-GUI-3B against 58.39% for Qwen2.5-VL-7B. It is the most direct competitor because its text summaries might preserve the evidence regardless of position. We run it with the released checkpoint and its own summary prompt on exactly the same episodes. To confirm our harness is fair, we first check that we reproduce roughly its reported GUI-Odyssey step success rate. For the long-horizon regime, the reference is LongHorizonUI, with a 77.3% average success rate on LongGUIBench against 65.8% for UI-TARS. We run its reflect-and-rollback loop under the same backbone where feasible, to test whether error recovery also removes positional bias. We expect it may not, since a confident stereotyped choice is not an execution error that reflection would flag. Executable Agentic Memory is excluded as a primary baseline. Its static graph memory assumes fixed screens, and our evidence content varies per episode.
For the decoding-level comparison, the reference is PCD as reported in Positional Contrastive Decoding: RULER F1 at 16k tokens rises from 64.40 to 69.04 for Llama-3-8B. Our implementation must reproduce a gain of that kind on RULER before we trust its screenshot results.
All methods share the same episodes, image resolution, action space and maximum history budget. Each learned component, our re-retrieval scorer included, gets the same hyperparameter-search budget on a held-out app. Beating the baselines means two things. First, the area under $G(p)$ across middle positions must shrink by a statistically significant margin, using paired bootstrap over counterfactual pairs. Second, the non-social $U(p)$ must be no worse. A method that lowers $G$ simply by refusing or randomizing does not count. The key ablation for our intervention is to keep the re-retrieval scorer but feed it a random past screenshot. This isolates whether the gain comes from recovering the specific evidence.
Why not just…?
Why not just use the accessibility tree instead of pixels? Text trees are exact, and OSWorld found they gave GPT-4 its best score. The same paper reports that these trees can run to millions of tokens, which pushes evidence deeper into the middle. OmniACT notes that many desktop apps lack such structure entirely. Rather than argue this, we include tree histories as a condition. If trees flatten $G(p)$, that is a real and useful finding, and we would report it as one.
Why not just tell the agent in its system prompt to ignore names and demographics? Uncovering Stereotypes in Large Language Models finds that guardrails weaken as tasks get more complex, and FairBelief locates biased beliefs at the model's default prediction. Both findings suggest an instruction cannot help when the agent has nothing better to go on. This remains a genuine threat, so we run a fairness-instruction arm. If it collapses $G(p)$ at all positions while $U(p)$ stays low, our memory framing is unnecessary for mitigation, though it would still explain where the bias comes from.
Why not just summarize the history into text notes? Summaries could carry the evidence forward intact. But a summarizer decides what matters at write time, before it knows the decision question. That is the same early-boundary commitment that caused cascaded losses in two-stage video pipelines (RGNet). The HAR baseline tests this directly. If HAR's curves are flat, re-retrieval offers nothing beyond it, and we would say so.
Why not just blame generic accuracy loss? If agents simply err more at middle positions, a gap could appear by chance. The stereotype fill-in rate rules this out. It conditions on errors and asks whether they lean toward the stereotyped direction, which a symmetric error pattern cannot produce.
Decisive Checks
Three results would settle the hypothesis. The first is the correlation, within each model, between $G(p)$ and $1-U(p)$ across positions, together with a fill-in rate significantly above the symmetric-error level. The second is a sharp jump in $G$ when the evidence crosses the sliding-window boundary. The third is that oracle re-presentation removes most of the middle-position gap while leaving the cue-free baseline unchanged. If all three hold, positional memory loss is a measurable cause of bias in screen-reading agents. If $G$ is flat across positions, the stereotype is a fixed prior, and memory-side fixes should be dropped from the bias agenda.
References
- LongHorizonUI: A Unified Framework for Robust long-horizon Task Automation of GUI Agent
- History-Aware Reasoning for GUI Agents
- Lost in the Middle: How Language Models Use Long Contexts
- Serial Position Effects of Large Language Models
- Mitigating Posterior Salience Attenuation in Long-Context LLMs with Positional Contrastive Decoding
- Uncovering Stereotypes in Large Language Models: A Task Complexity-based Approach
- FairBelief - Assessing Harmful Beliefs in Language Models
- AuditAgent: LLM Agent for Risks Auditing in Recommender Systems
- Can Large Language Model Agents Simulate Human Trust Behavior?
- AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents
- A Unified Framework and Dataset for Assessing Societal Bias in Vision-Language Models
- RGNet: A Unified Clip Retrieval and Grounding Network for Long Videos
- Learning To Locate Visual Answer In Video Corpus Using Question
- OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments
- Grounded Question-Answering in Long Egocentric Videos
- Executable Agentic Memory for GUI Agent
- OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist Autonomous Agents for Desktop and Web