TL;DR
LoopZip is a training-free visual token compression algorithm specifically designed for "looped" Vision-Language Models (VLMs). By accounting for how token importance shifts across repeated layer invocations, it achieves over 2x end-to-end speedups and a ~58% reduction in GPU memory while retaining nearly 97% of the original model's performance.
What Problem Does This Paper Address?
Vision-Language Models (VLMs) are getting deeper and more powerful, but this comes at a massive computational cost. A clever architectural workaround is the looped VLM, which reuses the same shared layers multiple times (like a recurrent network) to simulate a much deeper model with far fewer parameters.
However, there’s a catch: even if the parameters are fewer, the model still has to process long sequences of visual tokens (pixels turned into data points) every single time it loops through those shared layers. This leads to high latency and heavy GPU memory usage.
Existing token compression methods—which either prune "unimportant" tokens or merge similar ones—don't work well here. Why? Because in a looped model, a token that seems useless during the first pass might become critical during the third or fourth pass. Standard methods "forget" what the model might need later, leading to significant performance drops.
Methods
The authors introduce LoopZip, a three-stage, training-free pipeline that makes compression "loop-aware":
- Visual Aggregation: Before entering the looped backbone, LoopZip builds a compact candidate pool. It keeps the most salient patches (based on encoder attention) and merges the rest into "summary" tokens to ensure no information is completely discarded.
- Cross-Invocation Selection: This is the core innovation. Instead of picking tokens based on a single pass, LoopZip looks at the attention maps from the first two calls of a shared layer. By averaging this "instruction attention," it selects a set of tokens that are useful across different states of the model's reasoning process.
- Weight-Tied Value Correction: Removing tokens inevitably changes the output of a layer. LoopZip uses a mathematical "joint fitting" technique to adjust the values of the remaining tokens. This correction is "weight-tied," meaning it's designed to minimize the output deviation across multiple calls of the same shared layer simultaneously.
Datasets
The researchers evaluated LoopZip across two open-source looped VLMs (LoopVL-1B and SenseNova-Loop-9B-Preview) using eleven diverse benchmarks:
- Visual Understanding: MMStar, RealWorldQA, VMCB, AI2D, ChartQA, and HRBench4K.
- Hallucination Evaluation: POPE and HallusionBench.
- Multimodal Reasoning: MMMU-Pro, MathVision, and LogicVista.
Key Insights and Ideas
- The "Aha" Moment on Coverage: The authors found that looking at just the first call's attention only covered about 39% of the tokens needed in later calls. By simply averaging the first two calls, that coverage jumped to nearly 50%, significantly stabilizing the model.
- Stability through Correction: Even with the best tokens selected, removing the "background" tokens shifts the numerical output of the layers. The "Weight-Tied Value Correction" acts as a stabilizer, ensuring the compressed sequence produces an output as close to the full sequence as possible.
- Efficiency without Retraining: Because LoopZip is training-free, it can be plugged into existing looped VLMs without the massive cost of fine-tuning, making it highly practical for deployment.
Key Results
LoopZip sets a new state-of-the-art for VLM compression in looped architectures:
- Performance Retention: On LoopVL-1B, it retains 97.16% of the original performance at a 50% pruning ratio. At a more aggressive 87.5% pruning ratio, it outperforms the next best baseline (VisionZip) by 5.36%.
- Speed and Memory: When deployed on popular serving frameworks like vLLM and SGLang, LoopZip achieved 2.01–2.47× end-to-end speedups.
- Memory Footprint: It reduced peak allocated GPU memory by 58.36%–58.98%, allowing larger models to run on humbler hardware.
Limitations and Future Work
- Fixed Observation Window: The current method relies on the first two calls for selection. While ablations showed that adding more calls didn't provide monotonic gains, different architectures might benefit from a more dynamic observation window.
- Instruction Dependency: The selection process is guided by instruction attention. If an instruction is very short or vague, the compression might be less effective.
- Future Directions: The authors suggest exploring how LoopZip could be integrated with training-based methods or applied to even more complex "recursive" reasoning models that don't follow a fixed loop schedule.
Summary updated