SIEVE: Preserving Cross-Image Evidence via Depth-Aware Token Compression

Nitish Shukla, Surgan Jandial, Nhat-Tan Bui, Arun RossarXiv 2026 · arXiv

TL;DR

SIEVE is a training-free, "plug-and-play" framework for compressing visual tokens in multi-image Vision-Language Models (VLMs). By combining early spatial pooling with depth-aware, per-image token selection and merging, it achieves a 6.6× reduction in prefill FLOPs while actually outperforming uncompressed models on several multi-image benchmarks.

What Problem Does This Paper Address?

As Vision-Language Models move toward processing long sequences of images and videos, they hit a massive computational wall: the "KV-cache" and prefill FLOPs grow linearly (or quadratically) with the number of visual tokens. Existing token compression methods often fail in multi-image scenarios for three reasons:

  1. Global Starvation: Standard top-$k$ selection might keep 90% of tokens from one "interesting" image and discard a crucial detail from another image needed to answer the question.
  2. Identity Loss: Merging similar tokens across different images can confuse the model. If two images both contain a car, merging their tokens might make the model unable to count how many cars are present across the set.
  3. The Attention Lag: Most methods prune tokens in early decoder layers. However, the authors find that attention only becomes truly "informative" (query-aware) in the latter half of the decoder. Pruning too early risks throwing away the "needle" before the model even knows it's in a "haystack."

Methods

SIEVE (Spatial-Image-Evidence-depth-aware-VE) introduces a two-phase, training-free pipeline:

  • Phase 1: Pre-decoder Pooling: Before entering the decoder, each image is adaptively average-pooled into a compact grid. This keeps the initial processing cost low while maintaining the spatial layout.
  • Phase 2: In-decoder Selection & Merging: As the sequence passes through the decoder, SIEVE applies a multi-stage "select-and-merge" process at specific depths:
  • Equal Per-Image Quotas: Instead of a global top-$k$, SIEVE allocates an equal number of "slots" to every image, ensuring no image is entirely discarded.
  • Prompt-Attention Scoring: It uses the attention weights from the final prompt token to identify which visual tokens are most relevant to the query.
  • Same-Image Merging: Discarded tokens aren't just deleted; they are merged into the most similar retained token from the same image. This preserves information while strictly maintaining image identity.
  • Closed-Form Schedule: A geometric decay schedule determines exactly how many tokens to keep at each layer to hit a target "depth-averaged" token budget.
The SIEVE pipeline: pooling each image, then selecting and merging in stages over decoder depth.

Datasets

The authors evaluated SIEVE across four VLM backbones (LLaVA-OneVision, Qwen2.5-VL, Qwen3-VL, and InternVL3) using a wide array of benchmarks:

  • Multi-Image: Mantis-Eval, NLVR2, MuirBench, BLINK, and Q-Bench2.
  • Single-Image: TextVQA, ScienceQA-IMG, MMStar, and POPE.
  • Video: Video-MME and EgoSchema (treating frames as a sequence of images).

Key Insights and Ideas

  • Attention is a Late Bloomer: The authors demonstrate that in early layers, the model's attention is largely query-agnostic and suffers from positional bias. Query-relevant attention only peaks in the second half of the decoder.
  • The "Aha" Moment for Merging: Simply dropping tokens hurts performance. Merging them into "survivors" helps, but only if you restrict merging to the same image. Allowing cross-image merging drops accuracy by over 1 point because it destroys the model's ability to distinguish between distinct visual instances.
  • Budgeting Matters: By using a depth-averaged budget, SIEVE can afford to keep more tokens in the early layers (where they are cheap) and prune aggressively only once the attention signals become reliable.
Attention to queried objects peaks late (a), and the cumulative effect of SIEVE's components leads to significant gains (b) and preserved accuracy (c).

Key Results

  • Efficiency vs. Accuracy: On LLaVA-OneVision-7B, at a 12.5% token budget, SIEVE achieves a 65.13% average score, exceeding the uncompressed model's 63.12% while using 6.6× fewer prefill FLOPs.
  • Superiority over Baselines: At a tight 6.25% budget, SIEVE outperforms the strongest baseline (ToDRE) by 9.96 points on LLaVA-OneVision.
  • Robustness: In "distractor" tests (moving the relevant image among 7 irrelevant ones), SIEVE maintained significantly higher accuracy than global selection methods, proving the value of the equal per-image quota.
  • Video Performance: SIEVE leads on video benchmarks, showing it generalizes well to temporal sequences where "image identity" translates to "frame identity."

Limitations and Future Work

  • Training-Free Constraints: While being training-free is a major advantage for deployment, the authors note that the optimal number of stages ($S$) and the decay rate ($\gamma$) are currently set by heuristics. Future work could involve learning these parameters.
  • Computational Overhead of Scoring: While SIEVE reduces total FLOPs, the act of re-computing attention scores and performing similarity merges at multiple stages adds a small amount of latency compared to simple one-step pruning.
  • Extreme Compression: At extremely low budgets (e.g., <3%), even SIEVE begins to lose critical evidence, suggesting a floor for how much visual information can be compressed without loss.

Summary updated

Abstract

Multi-image vision-language models (VLMs) incur substantial computation when processing long visual sequences. Compressing these sequences must preserve complementary evidence across images and keep tokens from different images separate. Global selection can underrepresent question-relevant images, and pruning in early decoder layers risks removing relevant content before sufficient multimodal processing, yet existing methods spend most of the budget on these early layers. We propose SIEVE, a training-free, single-pass framework that jointly allocates visual tokens across images and decoder depth. SIEVE first pools each image into a compact spatial grid to reduce early processing cost. It then progressively selects representatives using prompt attention under equal per-image quotas and aggregates unselected tokens into similar representatives from the same image. These operations keep every image represented and its tokens separate from other images, while refining the retained representations. A closed-form schedule sets retention counts to target a prescribed average token budget across decoder layers. Across four VLM backbones, SIEVE achieves the highest average at a 6.25% budget on five multi-image benchmarks and four single-image benchmarks, and the highest mean on two video benchmarks. On LLaVA-OneVision-7B, it outperforms the strongest baseline by 9.96 and 9.03 points at 6.25% and 12.5% budgets, and at 12.5% it exceeds the uncompressed model by 2.01 points with 6.6x fewer prefill FLOPs.
BibTeX
@misc{shukla2026sieve,
  author = {Nitish Shukla and Surgan Jandial and Nhat-Tan Bui and Arun Ross},
  title = {{SIEVE: Preserving Cross-Image Evidence via Depth-Aware Token Compression}},
  year = {2026},
  eprint = {2611.00351},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.00351},
}
Figure