Vision-language models (a language model that reads images through a separate image encoder) see the world through a small, fixed-size window, typically a few hundred pixels on a side. When a 4K screenshot or a dense document is shrunk to fit that window, small text and tiny icons are destroyed, and a larger language model cannot recover them. The field has converged on two fixes. The first encodes many high-resolution crops, which is expensive. The second learns a policy that decides which crops or resolution to use. Today these policies are trained the way data augmentation policies were trained in 2018: every candidate choice is scored by running the full model on it. CARES labels the right resolution by sweeping the large model across several resolutions for 80K samples. ACTIVE-o3, ViGoRL and GUI-Spotlight train zoom policies with reinforcement learning, sometimes seeded by tree search with a 72B model. The augmentation literature escaped this cost by making the policy differentiable, which cut search from thousands of GPU hours to one or two.
This note proposes the same move for visual attention allocation, placed where it is actually sound. We do not push gradients through crop coordinates. Instead, we enumerate a fixed library of candidate views: global thumbnails at several scales plus a multi-scale quadtree of crops. We attach one relaxed gate to each view. Each gate enters the target model's attention as a log-bias on that view's tokens. With a single forward and backward pass over the full library, the answer's likelihood assigns credit to every view and every subset of views at once. A token-budget penalty pushes unneeded gates to zero. The per-sample gates learned this way become cheap training labels for a small predictor. At inference, that predictor sees only a low-resolution pass and the question, and decides which views to encode. The claim is specific: gradient-based gating recovers view-sufficiency labels that match an exhaustive rollout oracle, at a small fraction of the policy-search compute. It also produces a selector that dominates CARES-style global resizing and RL-trained cropping on the accuracy-versus-visual-tokens curve.
Core Claim
Current active-perception training treats the vision-language model as a black box. That forces either exhaustive enumeration, which is CARES's resolution sweep, or high-variance sampled search, which is GRPO/GSPO in ACTIVE-o3, ViGoRL and GUI-Spotlight. Both scale badly once the action is a subset of crops rather than one resolution, because the number of subsets grows exponentially.
The model is not actually a black box with respect to which of its input tokens it attends to. Suppose view $v$ contributes $n_v$ tokens and carries a gate $g_v \in (0,1)$. We add $\log g_v$ to the attention logits of every query attending to that view's tokens. At $g_v=1$ the view is fully present. As $g_v \to 0$ the view is exactly removed. Intermediate values interpolate smoothly. For image $x$, question $q$ and answer $y$, we optimize per-sample gate logits with the objective
$$ \min_{g}\; -\log p_\theta\!\left(y \mid x, q;\, g\right) \;+\; \lambda \sum_{v=1}^{V} g_v\, n_v , $$
where $p_\theta$ is the frozen target model, $V$ is the number of candidate views, and $\lambda$ trades accuracy against visual tokens. Following DADA, gates are sampled with a relaxed Bernoulli (Gumbel-sigmoid) distribution, and a straight-through hard gate is used late in optimization. This ensures the final labels describe genuinely discrete view subsets rather than soft blends the model never sees at inference.
Sweeping $\lambda$ gives each training sample a small family of subsets along its own accuracy-cost frontier. The approach differs from its sources in two ways. The augmentation papers relax the choice of image transformation and optimize one policy for a whole dataset. We relax which views of a single input are visible to a frozen model, per sample, and then amortize the result. It also differs from CARES, which chooses one global resize factor from exhaustive sweeps. We choose localized, multi-scale crop subsets from gradients.
Research Program
We build three things. The first is the view library. For each image we create global views at three scales and a quadtree of crops at two or three depths, and we add half-offset crops at each depth. The offset crops address the seam problem Mini-Monkey calls the semantic sawtooth effect, where fixed grid lines cut words and objects in half. Every crop carries row and column position embeddings in the spirit of UReader. The library is sized so that encoding all of it fits one training forward pass of the target model, roughly 15 to 25 views. The full-library cost is paid only during label generation, never at deployment.
The second is the gate optimizer, which produces labels. It runs a few dozen gradient steps on gate logits per sample, with the target model frozen and its key-value computations for the unchanged prefix cached. The optimizer runs over the same 80K TextVQA, ChartQA, DocVQA and LLaVA-Multi mixture CARES used. We add GUI screenshots and dense-scene data so that fine-grained grounding is represented. We will measure its cost against CARES's multi-resolution rollouts and against GRPO rollouts. The file reports no GPU-hour figure for either, so we will measure both ourselves at identical hardware.
The third is the amortized selector. It is a small vision-language model of the same size CARES uses (350M SmolVLM). It reads a low-resolution thumbnail and the question, then outputs an independent inclusion probability for every library view plus a budget-conditioning input $\lambda$. It is trained with binary cross-entropy against the hard gate labels. At inference only the selected views are encoded and sent to the target, so all savings occur upstream of tokenization. That is the side of the upstream-versus-downstream disagreement between CARES and Mini-Monkey that we take. We address Mini-Monkey's objection that upstream truncation destroys detail by selecting localized high-resolution crops rather than a global downscale.
One design tension from the file shapes the selector. Sample-Aware RandAugment and EntAugment show that per-sample uncertainty is a cheap and effective controller for augmentation strength. For resolution, however, the direction is inverted: hard, detail-dependent queries need more pixels, not gentler treatment. We therefore feed the target model's first-pass answer entropy to the selector as an input feature rather than hard-coding a rule. We also test the hard-coded rule as a baseline.
Baselines
The naive reference points are two fixed pipelines on the same target model, Qwen2.5-VL-7B, with a 72B replication for the headline table. The first sends the default single low-resolution view, the cheapest option. The second encodes the full view library, the most expensive option. Its accuracy is the ceiling any selector should approach. Every method is reported as a curve of accuracy against visual tokens and time to first token. A single point is not meaningful when budgets differ.
The strongest existing method for budgeted resolution choice is CARES. It trains a 350M proxy head on sufficiency labels from exhaustive multi-resolution rollouts. It reports a 70–80% reduction in prefill GFLOPs and visual tokens, up to 85% on documents, with a macro-averaged accuracy drop under 1% on Qwen2.5-VL-72B. We re-run it on identical training data and an identical proxy backbone. We also give it an extended action space, rollouts over our crop library, and we cap its label-generation compute at our gate optimizer's budget. Beating CARES means matching its token savings with a smaller accuracy drop, or greater savings at equal accuracy. The gap should widen on detail-heavy data (ScreenSpot-Pro, VisualOverload), where global resizing cannot help.
The strongest learned cropping policy is ACTIVE-o3, a GRPO-trained sensing module that proposes up to three boxes. It reports +8.5 APs on SODA-A and RealWorldQA 69.7 versus 67.9. We train it on our data with an equal GPU-hour budget, then compare at equal token counts. On GUI grounding, the reference numbers are GUI-Spotlight at 52.8% on ScreenSpot-Pro and ViGoRL at 23.7% on ScreenSpot-Pro and 86.4% on V*Bench. Both are multi-turn systems. We do not claim to beat their accuracy with one shot. The claim is a better accuracy-per-latency trade-off at comparable policy-training cost.
Three ablations isolate the key ingredient. The first replaces gradient-derived labels with labels from the target model's own attention mass on each view. The second uses leave-one-view-out likelihood drops as labels. The third removes the offset crops, to test whether seam coverage matters once selection is adaptive.
Why not just…?
Why not just read the view labels off the model's attention, as Mini-Monkey's token compression does? This is the most serious threat. Mini-Monkey shows that shallow-layer attention identifies redundant tokens with no training at all. Attention, however, measures where the model looks, not whether a view changes the answer. It also scores views independently, so it cannot see that two overlapping crops are redundant together. The attention-label ablation settles this directly. If attention labels match gradient labels on the oracle-agreement test below and on the final frontier, the gradient machinery is unnecessary and we would report that.
Why not just extend CARES's rollouts to crops? For a single global resolution, a sweep over three to five values is affordable. Over subsets of 20 views it is not. Greedy or leave-one-out rollouts reduce the cost but miss interactions between views. We run exactly this at matched compute. If a greedy rollout labeler equals our labels at equal GPU hours, the contribution shrinks to an engineering convenience.
Why not just gate zooming on answer entropy? It costs almost nothing and follows EntAugment. The rule is to encode the full library when first-pass entropy is high and stay at a thumbnail otherwise. It decides only whether to zoom, not where. It also depends on the low-resolution pass being uncertain when detail is missing, yet VisualOverload shows models often answer dense-scene questions confidently and wrongly (best counting accuracy 41.7%). We include this rule as a baseline and as a selector feature, and report its frontier separately.
Why not just keep RL, since GUI-Spotlight got strong results from only 18.5K samples? Small data is not small compute. GRPO/GSPO needs many sampled groups per example plus stabilizing scaffolding: dense heuristic rewards in ACTIVE-o3, and an auxiliary cross-entropy loss and SFT warm start in GUI-Spotlight. The matched-GPU-hour ACTIVE-o3 comparison measures whether our labels buy more accuracy per training hour.
Proposed Experiments
The decisive first check is oracle agreement. On 1,000 held-out samples with small libraries of eight views, we enumerate all 256 subsets with the target model. For each budget, this gives the true cheapest subset that still yields the correct answer. We then measure how often gradient gates, attention labels, leave-one-out and greedy rollouts recover a subset within one view of the oracle. We also record the compute each labeler spends. If gradient gates do not clearly beat attention and leave-one-out here, we stop and pivot.
The second experiment is the frontier study on TextVQA, ChartQA, DocVQA, OCRBench, V*Bench, ScreenSpot-Pro and VisualOverload. We plot accuracy against visual tokens and time to first token for all baselines. We expect the largest gains over CARES where detail is local, namely ScreenSpot-Pro and VisualOverload. We expect roughly parity on RealWorldQA, where a thumbnail usually suffices.
The third experiment is a policy-search cost audit. It reports wall-clock GPU hours for gate labeling, CARES rollouts and ACTIVE-o3 GRPO on the same hardware. The goal is to test whether the orders-of-magnitude savings DADA and Faster AutoAugment showed for augmentation carry over to view selection. We expect a smaller ratio, because each of our labels still requires a full-library forward pass.
The final check is the proxy-blindness failure. We bucket errors by whether the target object is visible in the selector's thumbnail. This tests whether the offset-crop library and entropy feature mitigate the under-allocation risk the file flags for CARES, or whether a small selector is simply the binding limit.
References
- ACTIVE-o3 : Empowering MLLMs with Active Perception via Pure Reinforcement Learning
- Grounded Reinforcement Learning for Visual Reasoning
- GUI-Spotlight: Adaptive Iterative Focus Refinement for Enhanced GUI Visual Grounding
- Differentiable Automatic Data Augmentation
- Mini-Monkey: Alleviating the Semantic Sawtooth Effect for Lightweight MLLMs via Complementary Image Pyramid
- UReader: Universal OCR-free Visually-situated Language Understanding with Multimodal Large Language Model
- Sample-Aware RandAugment: Search-Free Automatic Data Augmentation for Effective Image Recognition
- EntAugment: Entropy-Driven Adaptive Data Augmentation Framework for Image Classification
- CARES: Context-Aware Resolution Selector for VLMs
- VisualOverload: Probing Visual Understanding of VLMs in Really Dense Scenes
- Faster AutoAugment: Learning Augmentation Strategies Using Backpropagation