Abstract
Vision-language models (VLMs) typically process images at a native high-resolution, forcing a trade-off between accuracy and computational efficiency: high-resolution inputs capture fine details but incur significant computational costs, while low-resolution inputs advocate for efficiency, they potentially miss critical visual information, like small text. We present AwaRes, a spatial-on-demand framework that resolves this accuracy-efficiency trade-off by operating on a low-resolution global view and using tool-calling to retrieve only high-resolution segments needed for a given query. We construct supervised data automatically: a judge compares low- vs. high-resolution answers to label whether cropping is needed, and an oracle grounding model localizes the evidence for the correct answer, which we map to a discrete crop set to form multi-turn tool-use trajectories. We train our framework with cold-start SFT followed by multi-turn GRPO with a composite reward that combines semantic answer correctness with explicit crop-cost penalties. Project page: https://nimrodshabtay.github.io/AwaRes
Introduction
Modern Vision-Language Models (VLMs) face a significant challenge: high-resolution images are essential for tasks requiring fine-grained perception, such as reading small text in documents or identifying tiny objects in complex scenes. However, processing high-resolution images is computationally expensive, as the number of visual tokens typically scales quadratically with resolution. Existing efficiency methods, such as token pruning or full-image resolution escalation, often fall short—either by creating irregular patterns difficult to optimize in production or by wasting compute on irrelevant parts of the image.
We propose AwaRes, a "spatial-on-demand" framework that treats high-resolution acquisition as a tool-calling problem. By starting with a low-resolution global view and only requesting specific high-resolution crops when necessary, AwaRes achieves near-native high-resolution accuracy with a fraction of the computational cost.
The AwaRes Framework
The core of AwaRes is a multi-turn interaction loop. The model first processes a low-resolution version of the input image. Based on the query, it decides whether it has sufficient information to answer or if it needs a closer look. If more detail is required, the model emits a structured tool call (e.g., GET_CROPS:['center']) to retrieve high-resolution segments from a predefined set, such as quadrants or the center of the image.
This approach is inherently system-friendly. Unlike token pruning, it leverages standard multi-turn interactions and is highly compatible with KV-caching. The low-resolution tokens from the initial turn are reused in subsequent turns, minimizing redundant computation.
Automated Data Curation and Training
Training a model to know "where to look" without manual annotations requires a sophisticated data pipeline. We developed a three-stage automated curation process:
- LLM-as-a-Judge: A strong model (e.g., LLaMA-3.3-70B) compares answers generated from low-resolution and high-resolution versions of the same image to determine if a crop was necessary for correctness.
- Oracle Grounding: For cases requiring high-resolution, an oracle model localizes the specific visual evidence needed for the correct answer.
- Mapping: These localized regions are mapped to a discrete set of crops to generate training trajectories.
The model is trained in two stages. First, Cold-start SFT teaches the basic tool-calling protocol. Second, Multi-turn GRPO (Group Relative Policy Optimization) refines the policy. The RL phase uses a composite reward function that balances semantic accuracy with a penalty for unnecessary crop requests, effectively optimizing the efficiency-accuracy frontier.
Empirical Results
AwaRes was evaluated across six benchmarks spanning natural images and document/chart understanding. It achieved an average accuracy of 80.3%, nearly matching the 80.46% of the full high-resolution baseline. Remarkably, it reached this performance while using only 36% of the visual tokens on average.
In terms of speed, AwaRes demonstrates significant advantages. On the ChartQA benchmark, it was 7.7x faster than reasoning-heavy baselines like VisionThink, primarily because it avoids long intermediate reasoning traces in favor of direct, structured tool calls. Furthermore, AwaRes outperformed standard pruning methods like VisionZip by over 4% in accuracy at similar token retention levels.
Conclusion and Future Work
AwaRes demonstrates that "spatial-on-demand" processing is a powerful paradigm for efficient VLMs. By focusing high-resolution compute only where it matters, models can maintain high accuracy while significantly reducing latency and memory usage. Future work includes moving from a discrete crop set to continuous bounding box predictions and extending this spatial-on-demand approach to the temporal domain for efficient video understanding.
Summary updated