LoopZip: Visual Token Compression for Efficient Looped Vision-Language Models

Weizhong Huang, Zhicheng Liu, Yan Zhang, Fei Chao, Xiawu Zheng, Liujuan CaoarXiv 2026 · arXiv

TL;DR

LoopZip is a training-free visual token compression algorithm specifically designed for "looped" Vision-Language Models (VLMs). By accounting for how token importance shifts across repeated layer invocations, it achieves over 2x end-to-end speedups and a ~58% reduction in GPU memory while retaining nearly 97% of the original model's performance.

What Problem Does This Paper Address?

Vision-Language Models (VLMs) are getting deeper and more powerful, but this comes at a massive computational cost. A clever architectural workaround is the looped VLM, which reuses the same shared layers multiple times (like a recurrent network) to simulate a much deeper model with far fewer parameters.

However, there’s a catch: even if the parameters are fewer, the model still has to process long sequences of visual tokens (pixels turned into data points) every single time it loops through those shared layers. This leads to high latency and heavy GPU memory usage.

Existing token compression methods—which either prune "unimportant" tokens or merge similar ones—don't work well here. Why? Because in a looped model, a token that seems useless during the first pass might become critical during the third or fourth pass. Standard methods "forget" what the model might need later, leading to significant performance drops.

Methods

The authors introduce LoopZip, a three-stage, training-free pipeline that makes compression "loop-aware":

  1. Visual Aggregation: Before entering the looped backbone, LoopZip builds a compact candidate pool. It keeps the most salient patches (based on encoder attention) and merges the rest into "summary" tokens to ensure no information is completely discarded.
  2. Cross-Invocation Selection: This is the core innovation. Instead of picking tokens based on a single pass, LoopZip looks at the attention maps from the first two calls of a shared layer. By averaging this "instruction attention," it selects a set of tokens that are useful across different states of the model's reasoning process.
  3. Weight-Tied Value Correction: Removing tokens inevitably changes the output of a layer. LoopZip uses a mathematical "joint fitting" technique to adjust the values of the remaining tokens. This correction is "weight-tied," meaning it's designed to minimize the output deviation across multiple calls of the same shared layer simultaneously.
The LoopZip forward path showing aggregation, selection, and value correction.

Datasets

The researchers evaluated LoopZip across two open-source looped VLMs (LoopVL-1B and SenseNova-Loop-9B-Preview) using eleven diverse benchmarks:

  • Visual Understanding: MMStar, RealWorldQA, VMCB, AI2D, ChartQA, and HRBench4K.
  • Hallucination Evaluation: POPE and HallusionBench.
  • Multimodal Reasoning: MMMU-Pro, MathVision, and LogicVista.

Key Insights and Ideas

  • The "Aha" Moment on Coverage: The authors found that looking at just the first call's attention only covered about 39% of the tokens needed in later calls. By simply averaging the first two calls, that coverage jumped to nearly 50%, significantly stabilizing the model.
  • Stability through Correction: Even with the best tokens selected, removing the "background" tokens shifts the numerical output of the layers. The "Weight-Tied Value Correction" acts as a stabilizer, ensuring the compressed sequence produces an output as close to the full sequence as possible.
  • Efficiency without Retraining: Because LoopZip is training-free, it can be plugged into existing looped VLMs without the massive cost of fine-tuning, making it highly practical for deployment.
Comparison showing how combining two calls improves token coverage for later stages.

Key Results

LoopZip sets a new state-of-the-art for VLM compression in looped architectures:

  • Performance Retention: On LoopVL-1B, it retains 97.16% of the original performance at a 50% pruning ratio. At a more aggressive 87.5% pruning ratio, it outperforms the next best baseline (VisionZip) by 5.36%.
  • Speed and Memory: When deployed on popular serving frameworks like vLLM and SGLang, LoopZip achieved 2.01–2.47× end-to-end speedups.
  • Memory Footprint: It reduced peak allocated GPU memory by 58.36%–58.98%, allowing larger models to run on humbler hardware.
Figure 1. Performance and efficiency on LoopVL-1B. LoopZip maintains high performance under pruning while reducing infer- ence latency and GPU memory usage.

Limitations and Future Work

  • Fixed Observation Window: The current method relies on the first two calls for selection. While ablations showed that adding more calls didn't provide monotonic gains, different architectures might benefit from a more dynamic observation window.
  • Instruction Dependency: The selection process is guided by instruction attention. If an instruction is very short or vague, the compression might be less effective.
  • Future Directions: The authors suggest exploring how LoopZip could be integrated with training-based methods or applied to even more complex "recursive" reasoning models that don't follow a fixed loop schedule.

Summary updated

Abstract

Looped vision-language models (VLMs) deepen computation by reusing shared layers, approaching larger models' capabilities with fewer parameters and easing the storage and deployment burden of stacking independent layers. However, repeatedly processing long visual sequences still incurs substantial compute and GPU memory costs. Visual token compression reduces these costs by shortening sequences. Existing methods mainly select original tokens directly or combine selection with aggregation. In looped models, the former may overlook changing token importance across calls and discard information needed later; the latter lack joint consideration of output changes across calls, limiting information preservation. We propose LoopZip, a training-free visual token compression algorithm addressing both issues. Visual aggregation builds a compact candidate pool before the looped backbone, and cross-invocation selection averages instruction attention from the first two existing calls of the same shared attention layer to retain tokens needed by both states. Weight-tied value correction then jointly fits shared visual-value adjustments to both pre-selection layer outputs, reducing output deviations across states. To validate LoopZip's effectiveness, we conduct extensive experiments covering two open-source looped VLMs, eleven benchmarks, and three pruning ratios. On LoopVL-1B, LoopZip retains 97.16% of the original model's performance at 50% pruning ratio and exceeds the strongest baseline by 5.36% at 87.5% pruning ratio. Across Hugging Face Transformers, vLLM, and SGLang, LoopVL-1B with LoopZip reduces peak allocated GPU memory by 58.36%-58.98% and achieves 2.01-2.47x end-to-end speedups over the original model.
BibTeX
@misc{huang2026loopzip,
  author = {Weizhong Huang and Zhicheng Liu and Yan Zhang and Fei Chao and Xiawu Zheng and Liujuan Cao},
  title = {{LoopZip: Visual Token Compression for Efficient Looped Vision-Language Models}},
  year = {2026},
  eprint = {2611.22728},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.22728},
}
Figure