What Does "Now" Mean to Your Computer-Use Agent? Stale Screenshots, Page Loads, and Which Check to Run Before a Click

Vasundra SrinivasanarXiv 2026 · arXiv

TL;DR

Computer-use agents often fail because they plan their actions on "stale" screenshots of pages that are still loading. This paper reveals that the primary risk isn't the AI's planning time, but rather acting too quickly on a half-baked page; waiting for network activity to stop and performing a lightning-fast pixel check can eliminate the vast majority of these errors.

What Problem Does This Paper Address?

When an AI agent interacts with a website, it follows a loop: capture a screenshot, decide where to click, and then send that click to specific coordinates. The problem is a classic "Time-of-Check to Time-of-Use" (TOCTOU) bug. If the page layout shifts—perhaps because a slow-loading banner ad pops in or a font finally renders—the agent’s planned click might land on the wrong element.

Figure 1: One airline’s home page in a desktop Chrome Incognito window, in screenshots we took by hand on 28 October 2026; brand names are blurred. (a) A screenshot taken while the page loads: the search form is complete, and an agent planning on it would click the search button (box and dot; the dotted line marks its height). (b) The same page once it has loaded: a credit-card offer was inserted under the menu and pushed the form down 52 px, so the planned point lies on a checkbox that switches the search to loyalty points. The HTTP Archive’s crawl of this page on 1 October recorded the timing: a network-idle wait capped at 3 s ends at 3.5 s, the layout shifts 1.3 s later, and the network goes quiet only at 8.1 s. Rates across sites are in Section 5.1.

Prior research assumed the risk was proportional to how long the AI spent "thinking" (planning). This paper challenges that assumption, investigating exactly when screenshots become stale and how agents should verify the screen state before committing to an action.

Methods

The researchers took a multi-pronged approach to quantify and solve the staleness problem:

  1. Staleness Measurement: They tracked ~1,300 targets across 31 public web pages in three states: newly opened, recently scrolled, and fully loaded.
  2. Wait Rule Audit: They reimplemented the "wait rules" (how long an agent waits before taking a screenshot) of ten popular open-source agents to see how many stale clicks they allowed.
  3. Perception Comparison: They compared different ways to verify a target's position right before a click ("preflight checks"), including direct pixel comparison, OCR (Florence-2), small VLMs (Qwen3-VL, SmolVLM2), and frontier models (Claude 3.5 Haiku/Opus).
  4. Decision Theory: They derived a "break-even" formula to help developers decide if the latency cost of a pre-click check is actually worth the improvement in reliability.

Datasets

  • Real-World Pages: 31 public, read-only sites (clocks, maps, status pages) used for live tracking.
  • HTTP Archive Crawl: Data from 11.7 million desktop home pages (October 2026) to compare commercial booking/buying sites against .gov sites.
  • Preflight-Bench: A new, open-source benchmark generator that renders synthetic, moving web pages (e.g., market dashboards, news feeds) with exact ground-truth labels to stress-test agent perception.

Key Insights and Ideas

  • Loading, Not Planning, is the Culprit: On fully loaded pages, clicks remain valid for over 30 seconds. However, 17% of clicks planned in the first second after a page opens become stale. The "danger zone" is almost entirely during the initial page load.
  • Commercial Pages are Volatile: Booking and e-commerce sites are significantly "heavier" and more prone to layout shifts than government sites. 52% of commercial pages shifted their layout even after a 3-second wait.
  • Pixels for "Where," Models for "What": For spatial verification (is the button still at these coordinates?), simple pixel comparison is faster and more accurate than frontier VLMs. However, for semantic verification (did the payment actually go through?), 4B-parameter models perform as well as frontier models.
  • The Cost of Caution: Every check adds latency. The authors provide a formula ($p > p^*$) showing that if a planner is fast, the threshold for adding a check rises because a "stale" failure wastes less time.

Key Results

  • Wait Rules Matter: Waiting for "network idle" (no network requests for 500ms) reduced stale clicks to 0.46%, whereas eight out of ten existing agents used rules that left over 1% of clicks stale.
  • Pixel Check Efficiency: A 312ms pixel-based check detected 82% of stale clicks with only a 4% false alarm rate. In contrast, Claude 3.5 Opus had similar recall but a 12% false alarm rate and took 2.4 seconds.
  • Formula Accuracy: The proposed break-even formula correctly predicted whether to check or skip in 94.7% of test cases.
Figure 2: Left: share of planned clicks that remain valid on 31 public pages, by page state. Right: median wait of each rule after the page opens, and the share of clicks it leaves stale (95% CI). Only waiting for network activity to stop reaches 1%.

Limitations and Future Work

The authors acknowledge that their benchmark pages change much more frequently than real-world pages to act as a "stress test." Additionally, the commercial page analysis was based on a public crawl rather than interactive agent sessions, meaning checkout pages (which are often behind logins) were not measured.

Future work could involve measuring how these timing issues affect the training data for agents, as many pipelines record the "frame before the action" using fixed rules that might be capturing unstable, mid-load states.

Three Rules for Developers: 1. Wait for network activity to stop (up to ~3s) before the first screenshot. 2. Use a fast pixel check for spatial targets before irreversible actions. 3. If a check fails, re-plan from a new screenshot; do not try to "adjust" the old click coordinates.

Summary updated

Abstract

Computer-use agents plan on a screenshot for several seconds and then click at a position on the screen; if the page has changed in the meantime, the click lands on an unintended element. Prior work treats the planning interval as the source of this risk. Measuring planned clicks on 31 public web pages, we find that the risk lies in page loading: 0.4% of clicks planned on fully loaded pages became stale within 30 seconds, against 17% of clicks planned in the first second after a page opened. On these pages, waiting about one second for network activity to stop removes most of this risk, yet the wait rules of eight of ten widely used open-source agents leave more than 1% of clicks stale. Commercial pages keep loading for longer: in the HTTP Archive's public crawl, 15% of 65 desktop booking and buying home pages went 500 ms without network requests within 3 seconds of their HTML being parsed, against 63% of 3,028 .gov pages, and 52% shifted their layout within 6.5 seconds of a network-idle wait capped at 3 seconds, against 8% (61% against 26% among pages recorded in full). Where a check before the click is needed, as before irreversible actions and on pages that are still loading, a pixel comparison verifies the target's position as accurately as a frontier vision-language model at one eighth of the latency, and a 4B model matches the frontier model in verifying whether the previous action succeeded. We release a benchmark generator, a break-even formula for when a check pays, and three rules for agent developers: https://github.com/multimodal-ai-engineering/preflight-bench
BibTeX
@misc{srinivasan2026what,
  author = {Vasundra Srinivasan},
  title = {{What Does "Now" Mean to Your Computer-Use Agent? Stale Screenshots, Page Loads, and Which Check to Run Before a Click}},
  year = {2026},
  eprint = {2610.39282},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.39282},
}
Figure