Spending Screen Bits Where the Agent Is About to Look

Agents that operate software "by looking" read the screen as raw pixels and output mouse clicks or taps. They are usually tested on pristine, lossless screenshots. In deployment the screen often lives on a remote virtual machine, a cloud phone emulator, or a remote desktop session, and reaches the agent through a lossy video codec. A video codec (software such as H.264 or H.265 that compresses frames by throwing away detail) decides what to discard using models of human vision. Human vision barely notices a slightly blurred 12-pixel icon. A grounding model, which maps an instruction like "click the export icon" to an (x, y) location, can fail completely on it. This proposal makes two claims. The first is that this mismatch should be measured on GUI screens, which has not been done. The second is that the fix is cheap: the agent already sends an action to the remote machine at every step, so it can also send a one-sentence description of what it will look for next. A small encoder-side model then turns that text into a map of where to spend bits.

The opportunity sits between two literatures that do not cite each other. GUI agent work has moved away from structural metadata such as DOM trees and accessibility trees. It treats pixels as the universal interface, but it assumes those pixels arrive intact. Video-coding-for-machines work has learned to allocate bits for detectors and segmenters, but only on driving and surveillance video, with fixed task targets. GUI screens differ in two ways. Their statistics are synthetic: flat backgrounds, one-pixel edges, anti-aliased text. And which part of the screen matters changes with every instruction.

Core Claim

At a fixed per-step bit budget well below lossless size, a standard-compliant H.264/H.265 stream should do better on grounding accuracy if its per-block quantization map is conditioned on the agent's stated next intent. It should beat three alternatives: uniform rate control, an instruction-blind "protect edges and text" map, and an agent-driven zoom-and-request loop with the same total bits. The "quantization parameter" (QP) here is the per-block knob that sets how coarsely a 16×16 block is compressed.

The claim is falsifiable at three points. If lossless screenshot compression already fits realistic budgets, the problem is moot. If the instruction-blind map captures most of the gain, intent is unnecessary. If the zoom loop matches at equal bits and latency, encoder-side control is unnecessary. The experiments test each point directly.

Where This Sits Relative to Prior Work

Three papers motivate the gap from the GUI side.

  • ScreenSpot-Pro builds a benchmark of 4K/5K professional software where targets cover under 0.1% of the screen. It shows that downsampling destroys these targets. Its ScreenSeekeR planner-and-crop loop raises OS-Atlas-7B from 18.9% to 48.1%, but it assumes local access to full-resolution frames.
  • SeeClick and OmniParser both find icon grounding far more fragile than text grounding. That is exactly the high-frequency content a human-tuned codec removes first. OmniParser's detector, icon captioner, and text reader all depend on clean edges, and none of them is tested under compression.
  • ShowUI prunes 33% of visual tokens by grouping same-colored patches, which shows GUI screens are very redundant. However, it prunes inside the model after receiving uncompressed pixels. It never touches transmission, and its heuristic does not depend on the instruction.

The method comes from the coding-for-machines side.

  • Deep Video Codec Control for Vision Models predicts per-macroblock QP maps through a differentiable stand-in for the codec. It reports +20.5% segmentation accuracy over standard two-pass rate control under tight budgets.
  • RL-RC-DoT trains a block-level reinforcement-learning agent with a task reward. The downstream model is not needed at run time, and the agent costs about 4 ms per frame while saving 24.7% bitrate for car detection. We reuse its training recipe because it does not require differentiating through the grounder.
  • Firing Bits Where It Matters builds per-instance "just recognizable distortion" labels on COCO. These labels mark the coarsest compression at which a model still succeeds, and they show small objects need finer quantization. We build the GUI analogue.

All three coding-for-machines systems condition on a fixed visual task over natural video. We condition on language that changes at every step, and we work on synthetic screens.

One paper constrains the design. A Preprocessing Framework for Video Machine Vision under Compression reports that protecting features before quantization beats fine-tuning the downstream model on compressed data. That finding supports encoder-side intervention. It was obtained on Kinetics and tracking, so we re-test it on GUIs rather than assume it.

Research Program

Measurement. The first part is a measurement study and dataset, a GUI version of just-recognizable-distortion labels.

  • We re-encode every screenshot in ScreenSpot, ScreenSpot-Pro, and the AITW test split as single-frame and short-sequence H.264 and H.265 streams across a sweep of constant QPs.
  • We run three grounders on each version: ShowUI-2B, OS-Atlas-7B, and OmniParser with a GPT-4-class model.
  • For each instruction we record the coarsest QP at which the grounder still hits the target.
  • We analyze this threshold against element type (icon versus text), element area, and local contrast.
  • We relate each operating point to the lossless PNG size of the same screen.

This alone answers questions the field has not asked: how much bandwidth vision-only agents actually need, and which elements break first.

Method. The second part is an intent-conditioned allocator.

  • After each action, the agent emits a short text: what it expects to target next, and optionally a coarse region guess from the current frame.
  • On the remote machine, a small model takes a downscaled frame plus this text and outputs a ΔQP map (a per-macroblock offset to the encoder's quantization) for a stock encoder's region-of-interest interface. The stream remains standard-compliant.
  • The allocator only needs high recall at coarse resolution. It must keep the target and enough context to tell duplicate icons apart. It does not need pixel-precise localization, so it plays the cheap "planner" role from ScreenSeekeR rather than repeating the grounder.
  • Training follows RL-RC-DoT: PPO with reward equal to grounding success on the decoded frame minus λ times bits, with λ swept to trace a curve.
  • A supervised variant uses the threshold labels from part one as targets.
  • To test whether the allocator overfits to one grounder, we train it with ShowUI-2B's reward only and evaluate it unchanged with OS-Atlas-7B and OmniParser.

Baselines

Naive reference. The do-nothing reference is the stock encoder (x264 or x265) at the target bitrate with its default human-tuned adaptive quantization. Its partner is lossless PNG per step, which gives the accuracy ceiling and the bit cost any lossy scheme must undercut. For the uncompressed ceiling we use reported numbers. ShowUI-2B reaches 75.1% zero-shot on ScreenSpot (ShowUI). OS-Atlas-7B reaches 18.9% on ScreenSpot-Pro, and 48.1% with ScreenSeekeR (ScreenSpot-Pro). No reported result was found on Lacuna for any GUI grounder under lossy compression, so all compressed numbers will be produced here.

Strongest competitor. The strongest competitor is a bandwidth-accounted ScreenSeekeR-style pull loop. The client first sends a heavily compressed full frame. The agent-side planner names candidate regions, and the client returns those crops at high quality. Total bits are capped to the same budget, and the extra round trip is charged at a stated network latency. It is the strongest competitor because it already achieves the largest reported gain on the hardest benchmark, and it puts the same "spend detail where the instruction points" idea on the agent side.

Instruction-blind and oracle ablations. The instruction-blind learned allocator has the same architecture and training as ours, with the text input zeroed. It isolates the key ingredient. A hand-built edge-and-text map gives finer quantization to blocks with high gradient energy or detected text, and is the zero-training version of the same blind idea. An oracle map protects the ground-truth target box plus a fixed context margin, and serves as the upper bound on what intent can buy.

Fair comparison. All methods use the same encoder build, the same bitrate grid, and the same frozen grounders. Learned variants get identical compute and tuning budgets.

What counts as winning. Beating the baselines means a negative grounding BD-rate, the average bit saving at equal grounding accuracy integrated over the curve, relative to the pull loop at equal end-to-end latency. It also means a clear margin over the blind allocator on ScreenSpot-Pro icons.

Why not just…?

Why not just send a lossless PNG each step? GUI screens compress losslessly very well, and this is the most serious threat. If the measurement study shows lossless sizes within realistic per-step budgets for most ScreenSpot and AITW screens, we will report that plainly. The problem then narrows to the regimes where lossless fails. These are 4K professional screens containing rendered content such as CAD viewports, photos, and maps (ScreenSpot-Pro), cellular links, many concurrent agents per host, and video-rate control like the game setting in CRADLE. The size-versus-accuracy curves from part one decide which regime holds.

Why not just fall back to accessibility or DOM text? OmniACT documents that native desktop apps have no universal structural layer. On Mind2Web, OmniParser's pure-vision pipeline beat baselines with DOM access (OmniParser). Metadata is unavailable or worse exactly where pixels are needed.

Why not just fine-tune the grounder on compressed screenshots? Once a small icon's high frequencies are quantized to zero, there is nothing left to recover. On video tasks, pre-quantization protection beat fine-tuning (Preprocessing Framework). We will fine-tune ShowUI-2B on compressed data as an extra baseline. We expect it to help on text and large widgets but not on sub-0.1%-area icons. If it closes the gap there too, encoder control is unnecessary.

Why not just let the agent zoom (pull crops)? This is the strongest baseline above. It costs an extra round trip and spends bits on a context frame that the sender-side allocator can compress more selectively. When latency is low and budgets are loose, it may well tie. The sweep over latency and budget will show where each approach wins rather than declaring a single winner.

Proposed Experiments and Decisive Checks

There are four checks, run in order.

  1. Does the problem exist? On ScreenSpot-Pro with OS-Atlas-7B and ShowUI-2B, we look for the bitrate at which uniform encoding loses half its accuracy gap to lossless. We expect icons to break at much finer QP than text, mirroring the small-object result in Firing Bits Where It Matters.
  2. Does intent matter? We compare the intent-conditioned allocator, the blind allocator, the edge-and-text map, and the oracle map on grounding accuracy-versus-bits curves. If the blind version comes within a small fraction of the oracle's gain, we drop intent conditioning and report a GUI-tuned blind controller.
  3. Does encoder-side control beat agent-side zoom? We run the bandwidth-and-latency-accounted comparison against the pull loop across round-trip times.
  4. Does it hold beyond single steps and beyond one grounder? We test multi-step AITW episodes with a ShowUI-2B policy, scoring action-match versus bits per episode. We also measure transfer of the ShowUI-trained allocator to OS-Atlas and OmniParser. The success threshold is a grounding BD-rate saving that survives the switch of grounder, with allocator overhead small enough for interactive frame rates, in the few-millisecond range RL-RC-DoT reports for its controller.