TL;DR
CLASP introduces a training-free, plug-and-play framework that slashes the computational overhead of Multimodal Large Language Models (MLLMs) by dynamically pruning visual tokens based on the user's prompt. By adaptively fusing features across different Vision Transformer (ViT) layers and balancing token relevance with spatial coverage, CLASP achieves up to an 88.9% token reduction with minimal performance loss, delivering up to a 2.1× end-to-end inference speedup.
What Problem Does This Paper Address?
Multimodal Large Language Models (MLLMs) like LLaVA or Qwen-VL are powerful but computationally heavy. A massive bottleneck is the sheer number of visual tokens generated by vision encoders (like ViTs), which scales quadratically with image resolution. For instance, upgrading from LLaVA-1.5 to LLaVA-NeXT increases the token count from 576 to 2,880 per image, causing massive memory footprints and high latency.
Existing token reduction methods typically rely on features from a single, fixed ViT layer and apply static pruning rules regardless of the task. However, the authors show that this "one-size-fits-all" approach is brittle. A detail-oriented task like OCR or counting requires fine-grained, low-level features from shallower layers, while holistic scene understanding benefits from high-level semantic features from deeper layers. Static pruning fails to adapt to these diverse instruction intents.
Methods
The authors propose CLASP (Class-Adaptive Layer Fusion and Dual-Stage Pruning), a training-free, plug-and-play framework consisting of three main steps:
- Prompt-to-Class Routing: A lightweight, text-only router categorizes the incoming user prompt into a specific intent class (e.g., OCR, counting, scene understanding) before multimodal decoding begins.
- Class-Adaptive Layer Fusion: Instead of extracting visual tokens from a single ViT layer, CLASP dynamically blends features from multiple layers using category-specific weights. This ensures the model captures the optimal level of visual abstraction for the task at hand.
- Class-Adaptive Dual-Stage Pruning: The visual token budget is dynamically split based on the task category into two stages:
- Stage 1 (Relevance): Selects attention-salient "pivot" tokens that are highly relevant to the prompt.
- Stage 2 (Coverage): Uses a redundancy-aware spherical K-means clustering approach to select diverse "completion" tokens, ensuring the background and context aren't entirely lost.
An overview of this pipeline is illustrated below:
Datasets
The framework was rigorously evaluated across a comprehensive suite of benchmarks:
- Image Benchmarks: GQA, MMBench, MME, POPE, VQAv2, ScienceQA, TextVQA, and MMVet.
- Video Benchmarks: TGIF, MSVD, and MSRVTT (using Video-LLaVA).
No new datasets were introduced, but the authors used a small calibration subset (around 9k samples) to optimize the low-dimensional layer weights and split ratios offline.
Key Insights and Ideas
- Task-Specific Visual Granularity: The optimal feature depth depends heavily on the question type. Detail-centric tasks degrade when shifting focus to deeper ViT layers, whereas holistic tasks thrive on them.
- The Relevance-Coverage Trade-off: Attention-only pruning tends to select highly duplicated tokens from foreground objects, while similarity-only pruning can miss small, critical details (like text glyphs). CLASP's dual-stage approach elegantly balances both.
- Zero-Shot Generalization via Offline Calibration: Because the tuning parameters (layer weights and pruning ratios) are extremely low-dimensional, they can be optimized offline using a discrete subspace search on a small dataset without any MLLM finetuning, making it highly practical and robust.
Key Results
- Superior Token Reduction: On LLaVA-1.5-7B, CLASP consistently outperformed state-of-the-art methods like ToMe, FastV, and SparseVLM. It retained 94.7% of the original performance even under an aggressive 88.9% token reduction (shrinking 576 tokens down to just 64).
- Real-World Speedups: Efficiency analysis on the POPE benchmark showed that CLASP achieves a 1.5× to 2.1× end-to-end speedup while maintaining high accuracy, outperforming the baseline SparseVLM in both throughput and fidelity.
- Video Versatility: When extended to the video domain via Video-LLaVA, CLASP actually surpassed the unpruned upper bound average score (52.95 vs. 52.23), likely because it filters out massive temporal and spatial redundancy, acting as an effective noise filter.
- Qualitative Success: Visualizations demonstrate that CLASP robustly preserves critical regions (blue for attention-based foreground, red for similarity-based background context) across progressive pruning stages:
Limitations and Future Work
The authors note that the current routing policy relies on predefined text taxonomies and offline calibration. Future work will aim to transition from these manual heuristic priors to end-to-end learnable routing policies. Additionally, they suggest exploring tighter hardware-aware co-design to fully realize the latency benefits of token pruning on edge and embodied AI systems.
Summary updated