TL;DR
SIEVE is a training-free, "plug-and-play" framework for compressing visual tokens in multi-image Vision-Language Models (VLMs). By combining early spatial pooling with depth-aware, per-image token selection and merging, it achieves a 6.6× reduction in prefill FLOPs while actually outperforming uncompressed models on several multi-image benchmarks.
What Problem Does This Paper Address?
As Vision-Language Models move toward processing long sequences of images and videos, they hit a massive computational wall: the "KV-cache" and prefill FLOPs grow linearly (or quadratically) with the number of visual tokens. Existing token compression methods often fail in multi-image scenarios for three reasons:
- Global Starvation: Standard top-$k$ selection might keep 90% of tokens from one "interesting" image and discard a crucial detail from another image needed to answer the question.
- Identity Loss: Merging similar tokens across different images can confuse the model. If two images both contain a car, merging their tokens might make the model unable to count how many cars are present across the set.
- The Attention Lag: Most methods prune tokens in early decoder layers. However, the authors find that attention only becomes truly "informative" (query-aware) in the latter half of the decoder. Pruning too early risks throwing away the "needle" before the model even knows it's in a "haystack."
Methods
SIEVE (Spatial-Image-Evidence-depth-aware-VE) introduces a two-phase, training-free pipeline:
- Phase 1: Pre-decoder Pooling: Before entering the decoder, each image is adaptively average-pooled into a compact grid. This keeps the initial processing cost low while maintaining the spatial layout.
- Phase 2: In-decoder Selection & Merging: As the sequence passes through the decoder, SIEVE applies a multi-stage "select-and-merge" process at specific depths:
- Equal Per-Image Quotas: Instead of a global top-$k$, SIEVE allocates an equal number of "slots" to every image, ensuring no image is entirely discarded.
- Prompt-Attention Scoring: It uses the attention weights from the final prompt token to identify which visual tokens are most relevant to the query.
- Same-Image Merging: Discarded tokens aren't just deleted; they are merged into the most similar retained token from the same image. This preserves information while strictly maintaining image identity.
- Closed-Form Schedule: A geometric decay schedule determines exactly how many tokens to keep at each layer to hit a target "depth-averaged" token budget.
Datasets
The authors evaluated SIEVE across four VLM backbones (LLaVA-OneVision, Qwen2.5-VL, Qwen3-VL, and InternVL3) using a wide array of benchmarks:
- Multi-Image: Mantis-Eval, NLVR2, MuirBench, BLINK, and Q-Bench2.
- Single-Image: TextVQA, ScienceQA-IMG, MMStar, and POPE.
- Video: Video-MME and EgoSchema (treating frames as a sequence of images).
Key Insights and Ideas
- Attention is a Late Bloomer: The authors demonstrate that in early layers, the model's attention is largely query-agnostic and suffers from positional bias. Query-relevant attention only peaks in the second half of the decoder.
- The "Aha" Moment for Merging: Simply dropping tokens hurts performance. Merging them into "survivors" helps, but only if you restrict merging to the same image. Allowing cross-image merging drops accuracy by over 1 point because it destroys the model's ability to distinguish between distinct visual instances.
- Budgeting Matters: By using a depth-averaged budget, SIEVE can afford to keep more tokens in the early layers (where they are cheap) and prune aggressively only once the attention signals become reliable.
Key Results
- Efficiency vs. Accuracy: On LLaVA-OneVision-7B, at a 12.5% token budget, SIEVE achieves a 65.13% average score, exceeding the uncompressed model's 63.12% while using 6.6× fewer prefill FLOPs.
- Superiority over Baselines: At a tight 6.25% budget, SIEVE outperforms the strongest baseline (ToDRE) by 9.96 points on LLaVA-OneVision.
- Robustness: In "distractor" tests (moving the relevant image among 7 irrelevant ones), SIEVE maintained significantly higher accuracy than global selection methods, proving the value of the equal per-image quota.
- Video Performance: SIEVE leads on video benchmarks, showing it generalizes well to temporal sequences where "image identity" translates to "frame identity."
Limitations and Future Work
- Training-Free Constraints: While being training-free is a major advantage for deployment, the authors note that the optimal number of stages ($S$) and the decay rate ($\gamma$) are currently set by heuristics. Future work could involve learning these parameters.
- Computational Overhead of Scoring: While SIEVE reduces total FLOPs, the act of re-computing attention scores and performing similarity merges at multiple stages adds a small amount of latency compared to simple one-step pruning.
- Extreme Compression: At extremely low budgets (e.g., <3%), even SIEVE begins to lose critical evidence, suggesting a floor for how much visual information can be compressed without loss.
Summary updated