Spectral Evolution-Guided Token Pruning in Multimodal Large Language Models

Bin Chen, Yuxiang Cai, Yadan Luo, Yi Zhang, Jianwei Yin, Zhi ChenarXiv 2026 · arXiv

TL;DR

The paper introduces Cross-Layer Spectral Evolution (CLSE), a training-free visual token pruning framework that accelerates Multimodal Large Language Models (MLLMs). By tracking how token representations transform from high-frequency structural details to low-frequency semantic concepts across Transformer layers, CLSE accurately identifies and preserves critical visual information. This approach bypasses the positional biases of standard attention-based pruning, cutting FLOPs and KV cache memory usage in half with negligible impact on model accuracy.

What Problem Does This Paper Address?

When Multimodal Large Language Models (MLLMs) process images or videos, they convert visual data into hundreds or thousands of "visual tokens." This massive sequence length creates a severe computational bottleneck during the prefilling stage, leading to high inference latency and bloated Key-Value (KV) cache memory usage.

While many token reduction techniques exist, they typically rely on "instantaneous" signals from a single layer—such as raw attention weights or feature similarities. The authors point out two fatal flaws in this paradigm:

  1. Positional Bias: Standard attention mechanisms in LLMs often exhibit a strong bias toward tokens that appear later in a flattened sequence, regardless of their actual semantic value.
  2. Static Snapshots: Single-layer metrics ignore the dynamic, cross-layer transformations that visual tokens undergo as they fuse with text instructions.
Figure 1. Spectral evolution provides a more reliable token importance signal than attention. (a) Comparison between conventional text–vision attention and the proposed spectral evolution for estimating visual token importance across a Transformer block. (b) Layer- wise visualization shows that the attention is biased toward the visual tokens appearing later in the flattened sequence due to positional encoding effects, whereas spectral evolution consistently highlights semantically evolving regions across layers.

Methods

To overcome these limitations, the authors propose CLSE, a training-free, plug-and-play framework integrated into the early layers of the LLM decoder.

The core pipeline operates as follows:

  1. Frequency Domain Mapping: Visual tokens from a given layer are reshaped into a 2D spatial feature map and transformed into the frequency domain using a Fast Fourier Transform (2D FFT).
  2. High-Pass Filtering: A Gaussian high-pass filter is applied to suppress low-frequency global patterns, isolating the fine-grained, band-limited structural energy of each token.
  3. Evolution Intensity Calculation: The filtered representation is converted back to the spatial domain. CLSE then measures the normalized difference in this structural energy between adjacent layers (e.g., Layer 0 to Layer 1).
  4. Early Pruning: Tokens showing the highest "spectral redistribution" are deemed semantically active and retained, while static background tokens are dropped before reaching deeper, computationally heavy layers.

For video tasks, the authors introduce CLSE-M, which combines this spectral scoring with a token-merging strategy (via density-peak clustering) to recycle discarded tokens into compact representatives rather than dropping them entirely.

Figure 2. Overview of the proposed spectral evolution-guided token pruning framework. Left: Cross-layer spectral evolution is computed by transforming layer-wise token representations into the frequency domain, suppressing low-frequency components via a Gaussian high- pass filter, and measuring the normalized spectral difference between adjacent layers. Tokens exhibiting stronger spectral redistribution are considered decision-critical and preserved. Right: The CLSE module is integrated into early LLM decoder layers to guide token selection. Compared with raw attention scores, CLSE provides a more reliable signal for identifying semantically evolving tokens.

Datasets

The authors evaluated CLSE across a comprehensive suite of benchmarks:

  • Image Understanding & VQA: GQA, MMBench, MMBench-CN, MME, POPE (hallucination evaluation), ScienceQA, TextVQA, VizWiz, and OCRBench.
  • Video Question Answering: TGIF-QA, MSVD-QA, MSRVTT-QA, and ActivityNet-QA.

These evaluations were conducted across multiple prominent MLLM architectures, including LLaVA-1.5 (7B/13B), LLaVA-Next-7B, InternVL-2.5-8B, and Qwen2-VL (7B/72B).

Key Insights and Ideas

  • The Depth-Wise Spectral Shift: As visual tokens pass through an LLM, their representations naturally shift from high-frequency spatial details to low-frequency semantic abstractions. Tokens actively involved in text-vision reasoning exhibit a much more pronounced spectral redistribution across layers than redundant background tokens.
  • Mitigating Attention Bias: Because CLSE looks at structural changes across layers rather than absolute attention magnitudes, it remains entirely unaffected by the positional encoding artifacts that cause standard attention mechanisms to focus blindly on the end of a token sequence.
  • Linear vs. Quadratic Complexity: In modern inference engines utilizing optimized kernels like Scaled Dot-Product Attention (SDPA), extracting dense attention matrices forces a fallback to a quadratic $O(N^2)$ implementation. In contrast, CLSE's FFT-based scoring scales at a near-linear $O(N \log N)$ complexity, making it highly hardware-efficient.

Key Results

  • Performance Retention: On LLaVA-1.5-7B, CLSE retains 99.4% of the original model's performance while pruning 66.7% of the visual tokens (retaining 192 out of 576). Even under an aggressive 88.9% pruning ratio (retaining just 64 tokens), it maintains 94.8% accuracy, consistently outperforming baselines like FastV and PDrop.
  • Efficiency Gains: For image tasks, CLSE reduces total FLOPs to 43.9% and shrinks the KV cache from 313.0 MB down to 56.0 MB, delivering a 1.50× prefilling speedup.
  • Video Compression: On Video-LLaVA-7B, compressing video sequences from 2048 tokens down to 194 (a >90% reduction) using CLSE-M achieved 100% performance retention relative to the unpruned model, alongside a 2.73× prefilling speedup.
  • Scalability: The framework successfully scales to massive models like the quantized Qwen2-VL-72B, widening its performance lead over competing methods as pruning becomes more aggressive.

Limitations and Future Work

  • Surrogate Objective: The theoretical guarantees of CLSE are tied to a structure-preserving mathematical surrogate rather than a direct, task-specific semantic objective.
  • Hyperparameter Tuning: The framework relies on a manually set Gaussian cutoff ratio ($r=0.1$) to separate high and low frequencies, which might require tuning for radically different vision encoders.
  • Video Temporal Limits: For video tasks, the authors found that a 3D spatiotemporal FFT performed poorly due to sparse frame sampling ($T=8$). Consequently, CLSE relies on per-frame 2D FFTs combined with cross-layer dynamics, leaving explicit 3D spectral analysis for future work with denser video inputs.

Summary updated

Abstract

Reducing visual token redundancy is critical for accelerating Multimodal Large Language Models (MLLMs) without degrading cross-modal reasoning performance. Existing token pruning methods typically rely on single-layer signals, such as attention scores or token similarities, which overlook the cross-layer transformation of visual representations and may exhibit positional bias in multimodal token sequences. To address this limitation, we propose a training-free token pruning framework based on Cross-Layer Spectral Evolution (CLSE). Instead of measuring token importance from single-layer feature magnitudes, CLSE quantifies how token representations evolve across Transformer layers in the frequency domain. This evolution reflects the transition from high-frequency structural details to low-frequency semantic abstractions. We observe that tokens with stronger spectral redistribution across layers are more likely to be semantically active and should therefore be preserved. By modeling cross-layer token dynamics, CLSE provides a stable importance criterion that mitigates positional bias. Extensive experiments on both image and video benchmarks demonstrate that CLSE achieves a superior trade-off between efficiency and accuracy under aggressive token reduction. Across multiple MLLMs, CLSE reduces FLOPs, KV cache memory, and latency while maintaining competitive or improved performance.
BibTeX
@misc{chen2026spectral,
  author = {Bin Chen and Yuxiang Cai and Yadan Luo and Yi Zhang and Jianwei Yin and Zhi Chen},
  title = {{Spectral Evolution-Guided Token Pruning in Multimodal Large Language Models}},
  year = {2026},
  eprint = {2606.24165},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2606.24165},
}
Figure