Abstract
Multimodal Large Language Models (MLLMs) face significant computational overhead due to the high volume of visual tokens processed by Vision Transformers (ViTs) and Large Language Models (LLMs). Existing efficiency methods often overlook the ViT bottleneck or risk losing critical information through early pruning before text-context is available. We present IPCV (Information-Preserving Compression), a training-free framework that aggressively prunes visual tokens within the ViT while maintaining semantic integrity. IPCV utilizes Attention Stabilization (AS) to prevent feature distortion and a novel Neighbor-Guided Reconstruction (NGR) mechanism to restore pruned tokens before they reach the LLM. Evaluation across diverse image and video benchmarks shows that IPCV achieves significant computational savings—reducing latency to 50-70% of the original—while maintaining over 94% of vanilla performance, outperforming existing token reduction techniques.
Introduction
Multimodal Large Language Models (MLLMs) have achieved remarkable success in visual understanding, yet their deployment is hindered by the massive computational cost of processing visual tokens. High-resolution images and video sequences can generate thousands of tokens that must pass through both a Vision Transformer (ViT) and a Large Language Model (LLM).
Current efficiency research suffers from two primary limitations. First, many methods focus exclusively on LLM-side pruning, ignoring the significant bottleneck within the ViT. Second, early-stage ViT pruning is inherently risky; because it occurs before the model processes the user's text prompt, it may inadvertently discard small but critical details necessary for specific queries. Furthermore, the bidirectional nature of ViT attention means that removing tokens mid-stream can distort the features of the remaining tokens. IPCV addresses these challenges by combining efficient pruning with a novel reconstruction strategy that preserves the semantic interface for the LLM.
Methodology
IPCV is a training-free framework consisting of three distinct stages designed to maximize efficiency without sacrificing accuracy.
1. Early Pruning
Tokens are initially filtered in the shallow layers of the ViT. This selection is based on a feature-difference criterion, identifying redundant patches that contribute less to the overall representation of the image.
2. Attention Stabilization (AS)
To mitigate the "feature distortion" typically caused by sudden token removal, IPCV introduces Attention Stabilization. Pruned tokens are not immediately discarded; instead, for several subsequent layers, they continue to participate in the Attention mechanism as Keys and Values. Crucially, they skip the Feed-Forward Network (FFN) blocks. Since FFNs account for the vast majority of a ViT’s FLOPs, this approach provides necessary context to the surviving tokens at a negligible computational cost.
3. Neighbor-Guided Reconstruction (NGR)
NGR is the core innovation of the IPCV framework. The authors observed that tokens with high initial similarity tend to follow similar "trajectories" or evolutionary paths across transformer layers. At the final layer of the ViT, IPCV reconstructs the pruned tokens by taking their original state from the pruning layer and adding the average "delta" (the change from shallow to deep layers) of their nearest surviving neighbors. This ensures the LLM receives a complete set of tokens, preserving the original interface and semantic richness.
Experimental Setup
IPCV was evaluated across a comprehensive suite of benchmarks to ensure generalizability:
- Image Benchmarks: GQA, MMBench, MME, POPE, SEED, TextVQA, VizWiz, and OCRBench.
- Video Benchmarks: MVBench, EgoSchema, MLVU, and VideoMME.
- Models: The framework was tested on Qwen2-VL-7B and InternVL3-38B, highlighting its architecture-agnostic capabilities.
Results and Analysis
Performance Trade-offs
On the MMBench benchmark, IPCV demonstrated superior efficiency-accuracy trade-offs. With only 35% token retention, the model maintained 94.9% of its vanilla accuracy. In contrast, competing methods like ToMe or ToFu experienced much sharper performance degradation at similar compression levels.
Inference Speedup
IPCV significantly reduces total GPU inference latency, bringing it down to approximately 50-70% of the original model's time across various tasks. In video tasks (MVBench), IPCV achieved 95.7% of vanilla performance while operating significantly faster than methods that only prune tokens on the LLM side.
Resilience in Detail-Sensitive Tasks
Standard pruning often fails on tasks requiring fine-grained detail, such as Optical Character Recognition (OCR). However, IPCV’s reconstruction mechanism allows it to retain small textual details, leading to significantly better performance on OCRBench compared to traditional token-dropping methods.
Key Insights
- The "Delta" Consistency: The discovery that similar tokens evolve similarly through the network allows NGR to "predict" the final state of pruned tokens without executing expensive middle-layer computations.
- FFN-Centric Optimization: By identifying that FFNs are the primary source of computational cost, Attention Stabilization allows pruned tokens to provide context through attention blocks without incurring the heavy cost of the full transformer block.
- Orthogonality: Because IPCV outputs a full set of tokens to the LLM, it can be used in conjunction with LLM-side pruning methods (such as DART or FastV) for even greater cumulative speedups.
Conclusion and Future Work
IPCV provides a robust, training-free solution for accelerating MLLMs by addressing the ViT bottleneck. While NGR is an effective approximation, future work may explore more sophisticated reconstruction mechanisms to further reduce approximation errors for unique, non-redundant tokens. Additionally, dynamic stabilization depths based on input complexity could further optimize the balance between speed and precision.
Summary updated