TL;DR
Researchers have developed Glance-ETD, a framework that allows Multimodal LLMs (MLLMs) to retain over 91% of their performance while using only 5% of the original visual tokens. By using a "glance" mechanism to selectively encode high-resolution regions and a novel selection rule that balances scene diversity with question-relevant evidence, they achieved a 3.1× speedup on 4K image processing.
What Problem Does This Paper Address?
Multimodal Large Language Models (like GPT-4V or Qwen-VL) are computationally expensive, largely because high-resolution images are converted into thousands of "visual tokens." These tokens dominate the prefill stage—the initial phase where the model processes the input before generating a response.
Current methods for "token reduction" (pruning) face two major hurdles at extreme compression levels (e.g., keeping only 1 out of every 20 tokens):
- The Encoder Bottleneck: Even if you discard tokens before they hit the LLM, the vision encoder (like a ViT) still has to process every single patch of the image at high resolution, which is slow for 4K or 8K images.
- The Needle in the Haystack: When you prune 95% of tokens, you often lose the tiny details—like a phone number on a business card or a small street sign—that actually contain the answer to the user's question.
Methods
The authors propose a two-stage framework called Glance-ETD:
- Stage 1: The Glance Encoder: Instead of processing the whole image at native high resolution, the model first "glances" at a low-resolution version. A lightweight, question-conditioned scorer (only 1.9M parameters) identifies which regions are likely relevant. The model then re-encodes only those regions at full resolution. This drastically cuts down the vision encoder's workload.
- Stage 2: Evidence-Tempered Diversity (ETD): Standard pruning often uses "diversity" to ensure the kept tokens cover the whole image. However, diversity alone misses small details. ETD modifies the selection math: it takes the standard diversity score and "tempers" it with the evidence predicted by the scorer. This ensures the model keeps tokens that both represent the overall scene and focus on the specific answer.
To make the LLM comfortable with this "choppy" input, the authors use a single-rollout self-distillation recipe. The model is trained to match the outputs of a "teacher" model that has access to all the original tokens.
Datasets
The researchers evaluated Glance-ETD across 14 different benchmarks to ensure robustness:
- General Reasoning: GQA, MMMU, MMBench, MME, POPE.
- High-Resolution & Document Tasks: TextVQA, OCRBench, DocVQA, InfoVQA, HR-Bench (4K and 8K), MME-RealWorld, and HRScene.
- Models: While primarily tested on Qwen3.5-4B, they demonstrated generalization on Gemma 4 E4B-it, GLM-4.6V-Flash, and InternVL3.5-8B.
Key Insights and Ideas
- The "Where" Matters More Than the "What": At extreme compression, the selection strategy becomes the primary performance driver. Training the model to be robust to missing tokens helps, but if the selector drops the "answer tokens" entirely, no amount of training can recover the information.
- Encoder Cost is Quadratic: For high-resolution images, the vision encoder's self-attention becomes the dominant cost. By only looking at 25% of the image at high resolution, Glance-ETD bypasses the quadratic scaling of global attention.
- Diversity vs. Relevance: The paper highlights a beautiful trade-off. Pure diversity (DivPrune) is great for counting objects or describing layouts. Pure relevance (Top-K) is great for reading text but loses the context. ETD is the "Goldilocks" solution that blends both.
Key Results
- Performance Retention: At a strict 5% token budget, Glance-ETD retains 91.6% of the full-token performance.
- Superiority over SOTA: It outperforms the previous best method, LT-OPD, by a massive 17.3 points on average.
- Speed & Efficiency:
- 3.1× faster Time to First Token (TTFT) on 4K images.
- Reduces vision encoder FLOPs to roughly 1/6th of the original.
- Significant memory savings, allowing for larger batch sizes during inference.
- Text-Rich Dominance: The biggest gains were seen in document-heavy tasks (DocVQA, OCRBench), where identifying specific small regions of text is critical.
Limitations and Future Work
- Box Supervision: The scorer currently requires training on datasets with bounding box annotations (like TextOCR or Visual-CoT) to learn what "evidence" looks like. While it generalizes well, a purely self-supervised scorer would be a powerful next step.
- Extreme Budgets: While 5% is impressive, performance still drops compared to the full-token model. There is still a "compression tax" for complex reasoning.
- Video Extension: The authors suggest that this "glance and select" logic is naturally suited for video, where temporal redundancy is even higher than spatial redundancy, though this wasn't explored in the current paper.
Summary updated