Agents that operate a computer from screenshots alone, without reading the page's code or the operating system's accessibility tree, hit a hard limit on professional software. A 4K CAD or audio-editing screen has to be shrunk before a vision-language model can read it. When that happens, a ten-pixel toolbar icon becomes a few gray pixels, and no amount of reasoning can recover it. Current fixes either run the model several times on zoomed-in crops, which multiplies latency, or train the model to be more consistent across zoom levels, which, as its own authors note, cannot restore information the downsampled image no longer contains. We propose a third route, taken from image fusion, a field whose central problem is combining a smooth base image with fine detail without washing the detail out. The idea is to keep the downsampled screen as the model's main input, compute the fine edge detail that downsampling throws away from the full-resolution pixels, and fold it back into each visual token through a lightweight side channel. The model still reads the screen once with a fixed number of tokens, but each token now carries a compact summary of the strokes and outlines beneath it.
Put in two sentences: today's grounding models see a blurred screen because shrinking an image works like a low-pass filter that deletes exactly the high-frequency contours that make up icons and small text. We recover those contours cheaply with a wavelet split at full resolution and inject them into the tokens, borrowing the "bypass the bottleneck" design and the "never average fine detail" rule from detail-preserving image fusion.
Core Claim
At a fixed visual-token budget and a single forward pass, adding a full-resolution high-frequency side channel to each visual token raises grounding accuracy on tiny targets. It does better than three alternatives at the same budget: spending those tokens on higher uniform resolution, reallocating resolution toward busy regions, and training-time zoom regularization. The gains should be concentrated on icons and on targets below about 0.1% of screen area, and accuracy on ordinary screens should not drop. If this holds, the sub-pixel limit described in Spatially Stable GUI Grounding via Zoom Consistency Loss is a limit of the input representation, not of single-pass grounding.
Where This Sits
Two lines of work motivate the proposal. The first is the multi-pass zoom family, which shows that fine detail matters a great deal. ScreenSpot-Pro introduces a benchmark of professional applications and the ScreenSeekeR planner–grounder, which raises OS-Atlas-7B from 18.9% to 48.1% by grounding on crops. AdaZoom-GUI adds a gated second pass on a crop when the first predicted box is small. GUI-Spotlight has the model call crop tools repeatedly. All three process global context and local detail in separate calls and never merge them into one representation. AdaZoom-GUI also gives the clearest evidence that crops alone are not the answer: always zooming drops ScreenSpot-v2 accuracy from 0.945 to 0.916, because the crop loses layout context. We keep the global view and add detail to it, rather than trading one for the other.
The second motivating line is single-pass regularization. Zoom Consistency Loss trains Qwen3-VL-8B so that its full-screen prediction agrees with a zoomed teacher, reaching 57.31 on ScreenSpot-Pro in one pass. It explicitly concedes that the method fails when the target vanishes in the downsampled input. Our work answers that concession directly, and the two methods can be combined.
The closest token-level work constrains the design. ShowUI prunes about a third of visual tokens by grouping patches with near-identical RGB values. It decides which tokens to keep, but it does nothing to change what a kept token knows about sub-patch detail. UGround slices high-resolution screens into tiles, which preserves detail but makes token count grow with screen area.
The method comes from image fusion. WaveGAN splits features into Haar wavelet bands: one smooth base band plus horizontal, vertical and diagonal detail bands. It routes the detail bands around the network's compressing bottleneck through skip connections. It also reports that simply averaging detail bands across inputs (its WaveGAN-M variant) gets worse as inputs multiply, because slightly shifted edges cancel. CRNet separates high frequencies cheaply by contrasting max pooling against average pooling, and uses large depth-wise convolutions as learned high-pass filters, at modest memory cost. Classical fusion work on NSCT-domain fusion and spatial-domain averaging adds two more constraints. Long-support wavelets ring around sharp edges, and linear averaging cuts contrast. Screens are almost entirely sharp edges, so we use the shortest wavelet (Haar). We never modify the pixels the encoder sees, and we pool detail with a magnitude-preserving rule instead of a mean.
Research Program
Let $I$ be the full-resolution screenshot and $s$ the downsampling factor needed to fit the token budget $B$. The base model, Qwen3-VL-8B (chosen so results line up with the zoom-consistency paper), encodes the downsampled image $I_{\downarrow s}$ as usual.
In parallel, we run $\log_2 s$ levels of Haar decomposition on $I$. The smooth band at the last level is close to what the encoder already sees. The detail bands at every level are exactly what downsampling discards. For each encoder patch $p$, we collect the detail coefficients under its footprint and pool them with a signed max-magnitude rule per band and channel. This keeps a one-pixel outline at full strength instead of diluting it into its flat surroundings, and it avoids the phase cancellation WaveGAN-M exhibits. A small convolutional detail encoder, $g$, maps these pooled bands to a vector of the same width as the vision tokens. It follows CRNet's pooling split and large-kernel depth-wise design, has a few million parameters, and costs negligible compute next to the language model.
The fused token is
$$t_p = f(I_{\downarrow s})_p + \alpha_p \, g(D_p),$$
where $f$ is the frozen-then-finetuned vision encoder output for patch $p$, $D_p$ is the pooled detail tensor for that patch, and $\alpha_p$ is a learned per-token gate initialized at zero. Starting from zero means training begins exactly at the base model, and the gate learns to stay near zero on flat regions.
We train with the standard click-coordinate loss on the 70.7K-sample Salesforce grounding mix used by the zoom-consistency paper. One variant adds an auxiliary loss: from $t_p$ alone, a small head must reconstruct the full-resolution detail bands of the patch. This is a direct analogue of WaveGAN's frequency L1 loss, and it forces the tokens to actually carry the detail.
The program has three parts. First, a controlled synthetic study renders real application icons and labels at shrinking sizes onto realistic screen backgrounds. It measures the size at which grounding collapses, with and without the side channel, at several token budgets. Second, the main benchmark study is described below. Third, an analysis checks whether the tokens really contain sub-patch information, using linear probes that predict icon identity for targets smaller than one patch.
Baselines
The naive reference point is Qwen3-VL-8B fine-tuned on the same 70.7K samples, with the same schedule, at the same token budget, and no side channel. Every claimed gain is measured against this model first.
The strongest single-pass competitor is Zoom Consistency Loss on the same base and data. Its reported 57.31 on ScreenSpot-Pro (Zoom Consistency Loss) is the figure to beat for one-pass inference. We will reproduce it in our pipeline, because the file does not state its standalone baseline. We also run side channel plus consistency loss together, since the two address different failures.
The compute-matched resolution baseline gives the plain model more tokens. We run it at budgets $B$, $2B$ and $4B$, up to native resolution where memory allows. This tests whether the side channel is a cheaper substitute for resolution. Beating it means our model at $B$ matches or exceeds the plain model at $2B$.
A detail-guided allocation baseline uses the same Haar energy map to give busy regions finer tiles and flat regions coarser ones, at total budget $B$. Without this baseline, a reviewer could argue that a ShowUI-style redistribution captures the same benefit.
The strongest multi-pass systems are references, not like-for-like competitors. AdaZoom-GUI-4B reports 76.8% on ScreenSpot-Pro, but that relies on a 397B instruction refiner. Without the refiner it reports 70.6% (AdaZoom-GUI). GUI-Spotlight reports 52.8% and ScreenSeekeR 48.1%. We report accuracy against measured latency and FLOPs per action. We also apply AdaZoom's gated second pass to our model, to test whether the two stack.
Ablations isolate the key ingredient. One replaces max-magnitude pooling with mean pooling, which tests the fusion-derived rule. One feeds $g$ the downsampled image instead of full-resolution detail, which checks whether a few added parameters, not new information, explain any gain. One removes the reconstruction loss. One shuffles side-channel inputs across screenshots at test time.
Why not just…?
Why not just run the model at native resolution? This is the most serious threat. Modern vision-language models accept dynamic resolution, and given enough tokens they would see everything. The answer is cost. ScreenSpot-Pro screens average 3267×1727, and token count grows with area, while ShowUI and Aguvis both argue that compact visual inputs matter for agent cost. The budget sweep settles the question. If the plain model at $2B$ beats our model at $B$, the side channel is not worth it and we will say so. We also report the native-resolution ceiling where it fits.
Why not just zoom in a second pass, like AdaZoom-GUI? Multi-pass zooming works and currently holds the best reported numbers. However, it doubles forward passes per step, depends on the first guess landing in the right region (GUI-Spotlight describes this blind-spot failure), and loses context when overused, as shown by the 0.945 to 0.916 drop. Our claim is not that one pass beats two. It is that one detail-carrying pass closes much of the gap at half the cost, and that the second pass is gated more accurately when the first pass is better. The accuracy-versus-latency comparison and the stacked variant test both claims.
Why not just sharpen the screenshot or spend tokens where the detail is? Pre-sharpening with a classical transform cannot put back pixels that downsampling has merged, and longer wavelets ring on sharp UI edges (NSCT-domain fusion). We include a sharpen-then-downsample control to confirm this. Detail-guided reallocation is a real alternative and is run as a full baseline. If it matches the side channel, the useful insight is "use a frequency map to guide token placement," and we will report that result instead.
Proposed Experiments and Decisive Checks
The main evaluation is ScreenSpot-Pro, broken down by application group, by text versus icon target, and by target-area bins. ScreenSpot-v2 checks that ordinary screens do not degrade, with AdaZoom's always-zoom 0.916 as a cautionary reference against the 0.945 gated result. OSWorld-G and MMBench-GUI-L2 test transfer; the zoom-consistency paper reports gains of +1.60% and +1.51% on these, but no absolute scores are found in the file.
Three results would decide the question. The first is the size knee in the synthetic study: the side channel should move the collapse point to smaller targets, and mean pooling should give back part of that shift. The second is the shuffle test: swapping detail inputs across screenshots should bring accuracy back to the plain baseline, which proves the model uses the channel. The third is the budget curve: the proposal succeeds if our model at budget $B$ beats the reproduced 57.31 single-pass result, matches the plain model at $2B$ or more, and concentrates its gains on icons and the smallest area bins. It fails if the gains are uniform across sizes, which would point to extra capacity rather than recovered detail, or if detail-guided reallocation does just as well.
References
- Spatially Stable GUI Grounding via Zoom Consistency Loss
- ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Use
- AdaZoom-GUI: Adaptive Zoom-based GUI Grounding with Instruction Refinement
- GUI-Spotlight: Adaptive Iterative Focus Refinement for Enhanced GUI Visual Grounding
- ShowUI: One Vision-Language-Action Model for GUI Visual Agent
- Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents
- WaveGAN: Frequency-Aware GAN for High-Fidelity Few-Shot Image Generation
- CRNet: A Detail-Preserving Network for Unified Image Restoration and Enhancement Task
- Log-Gabor Energy Based Multimodal Medical Image Fusion in NSCT Domain
- MODIFIED PRIMITIVE IMAGE FUSION TECHNIQUES FOR THE SPATIAL DOMAIN
- Aguvis: Unified Pure Vision Agents for Autonomous GUI Interaction