Mechanistic Insights into Functional Sparsity in Multimodal LLMs via CoRe Heads

Ruoxi Sun, Quantong Qiu, Juntao Li, Zecheng Tang, Yihang Lou, Min ZhangarXiv 2026 · arXiv

TL;DR

Multimodal Large Language Models (MLLMs) exhibit a profound structural property called "functional sparsity," relying on a tiny fraction of specialized attention heads—termed Context-aware Retrieval (CoRe) heads—to extract task-relevant visual information. By surgically preserving these critical heads while applying sparse attention to the rest, researchers achieved up to a 2.3× inference speedup during the prefill stage with virtually no loss in downstream reasoning performance.

What Problem Does This Paper Address?

While MLLMs excel at complex vision-language tasks, the internal mechanics of how they filter out background noise to retrieve query-relevant visual features have remained a black box. Previous interpretability frameworks relied on coarse statistical heuristics (like spatial entropy) evaluated on simple, static images. These approaches fail to explain how models handle fine-grained, query-conditioned attention in dense, cluttered, or spatiotemporal (video) scenes. Understanding these internal retrieval mechanisms is essential for diagnosing model failures and optimizing inference efficiency.

Methods

The authors introduce a token-level metric called Retrieval Attention Mass (RAM) to quantify how much attention a specific head allocates from instructional query tokens to semantically relevant visual tokens.

Using RAM, they isolate Context-aware Retrieval (CoRe) heads—a highly specialized subset of attention heads that act as dedicated information extractors.

To turn this mechanistic insight into a system-level optimization, they propose the CoRe-head-Guided Hybrid Attention Paradigm:

  1. Top-$k$ Full Attention: The highest-ranked CoRe heads retain standard, global dense attention to preserve their precise cross-modal extraction capabilities.
  2. Stream Sparse Attention: The remaining non-essential heads are restricted to a localized sliding window attention pattern, drastically reducing computational complexity from $O(N^2)$ to $O(N \cdot w)$.
Overview of the CoRe head probing pipeline

Datasets

The authors evaluated CoRe heads across four diverse visual domains to ensure cross-task generalizability:

  • RefCOCOg: Object-level visual grounding on static images.
  • VidSTG: Video-based spatio-temporal grounding.
  • MMLongBench: Long-context multimodal reasoning across massive visual sequences.
  • MMDocIR: Document layout understanding and document-grounded retrieval.

System-level acceleration was benchmarked using the MLVU (long video understanding) and Video-MME benchmarks.

Key Insights and Ideas

  • Functional Dichotomy: MLLM attention heads naturally divide labor. CoRe heads act as precise, localized information extractors, while standard vision heads facilitate global feature aggregation.
  • The Deep-Layer Bottleneck: As model capacity scales (e.g., from 4B to 32B parameters), CoRe heads spontaneously converge from being scattered across middle layers into a highly concentrated, contiguous block within the deep layers (layers 44–56 in Qwen3-VL-32B). Larger models naturally develop a narrower architectural bottleneck for cross-modal integration.
  • Globally Consistent, Locally Variant: Across entirely different datasets and modalities, a core set of approximately 30 attention heads remains universally activated, while the remaining CoRe heads dynamically adapt to specific data distributions.
  • Implicit Regularization: Restricting non-essential heads to localized sliding windows actually improves performance on certain precise spatiotemporal tasks. This suggests that sparse attention acts as a regularizer, filtering out distracting cross-modal background noise.
Figure 1: Functional specialization in MLLM at- tention heads on RefCOCOg. Left (CoRe Heads): High-attention regions correspond to context- relevant objects. Right (Bottom Heads): High- attention regions show week context-relevant.

Key Results

  • Causal Necessity: Masking just the top 5% of CoRe heads causes a catastrophic collapse in multimodal reasoning performance (e.g., Qwen's accuracy on MMLongBench plummets from 45.3% to 27.0%). Conversely, ablating the bottom-ranked heads yields almost no performance degradation.
  • Information Concentration: The top 15 CoRe heads alone capture up to 25.6% of critical visual tokens, whereas the bottom 15 heads capture a negligible fraction (~2%).
  • Inference Acceleration: The CoRe-guided hybrid attention mechanism achieves substantial prefill speedups—up to 2.3×—while maintaining or slightly exceeding the accuracy of fully dense baselines on long-video benchmarks.
Figure 2: Mechanistic evidence of functional specialization in MLLMs attention heads on VidSTG. Left (CoRe Heads): A sparse subset of specialized heads acts as precise information extractors, surgically isolating context-relevant entities (e.g., “red car”, “adult in white”) by filtering background noise across key frames. Right (Bottom Heads): The vast majority of heads exhibit semantic dispersion, scattering attention across irrelevant regions and failing to ground the instruction.

Limitations and Future Work

The primary limitation of the RAM metric is its heavy reliance on explicit, ground-truth spatial or temporal annotations (such as bounding boxes or video tubes) to locate target visual entities. This makes it difficult to identify CoRe heads in entirely unannotated datasets or purely abstract reasoning contexts. Furthermore, the study focused exclusively on late-fusion transformer architectures; whether this functional sparsity emerges in natively early-fusion multimodal models remains an open question. Finally, the authors suggest that moving from their current static head configuration to a dynamic, input-adaptive routing strategy is a promising direction for future work.

Summary updated

Abstract

While Multimodal Large Language Models (MLLMs) demonstrate remarkable proficiency on complex vision-language tasks, the mechanisms by which they extract query-relevant visual features from complex, noisy contexts remain opaque. In this paper, we present an in-depth interpretability study that uncovers a profound structural property within MLLMs: functional sparsity in cross-modal retrieval. Leveraging a token-level metric termed Retrieval Attention Mass (RAM), we identify and characterize a highly specialized subset of attention heads, referred to as Context-aware Retrieval (CoRe) heads. Across diverse visual domains and model scales, we observe a clear functional division: CoRe heads act as dedicated information extractors, while most other heads distribute attention over broader contextual regions. Causal interventions further demonstrate the necessity of these specialized heads. Ablating only the top 5% of CoRe heads causes significant degradation in multimodal reasoning performance, whereas ablating lower-ranked heads has minimal effect. Moreover, acceleration experiments validate the utility of CoRe heads, showing that leveraging this localized sparsity significantly accelerates inference while maintaining robust task performance. Our findings reveal a structural principle of functional sparsity within MLLMs, refining the current understanding of mechanistic interpretability and laying a theoretical foundation that can inspire future architecture design and model optimization.
BibTeX
@misc{sun2026mechanistic,
  author = {Ruoxi Sun and Quantong Qiu and Juntao Li and Zecheng Tang and Yihang Lou and Min Zhang},
  title = {{Mechanistic Insights into Functional Sparsity in Multimodal LLMs via CoRe Heads}},
  year = {2026},
  eprint = {2606.05843},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2606.05843},
}

Loading related content...

Figure