Sparse Shortcuts: Facilitating Efficient Fusion in Multimodal Large Language Models

Jingrui Zhang, Yong Zhang, Feng Liang, Wei Wang, Runhao Zeng, Xiping HuSubmission to ICLR 2026 · Rejected

TL;DR

SparseCut is a novel architecture for Multimodal Large Language Models (MLLMs) that bridges the gap between vision and language by introducing "sparse shortcut" connections between intermediate layers of the vision encoder and the LLM. By using a U-shaped connection pattern and a clever multi-grained fusion module, it captures fine-grained visual details without the quadratic computational explosion typically associated with high-resolution image processing.

What Problem Does This Paper Address?

Most current MLLMs (like the popular LLaVA framework) follow a "bottleneck" design: they take only the final output layer of a vision encoder (like CLIP) and feed it into the LLM. This approach has two major flaws:

  1. Loss of Detail: Vision Transformers (ViTs) are hierarchical. Early layers capture textures and edges, while later layers capture global semantics. By only using the last layer, the model "forgets" the fine-grained details necessary for complex spatial reasoning.
  2. The High-Res Tax: To see more detail, researchers often increase image resolution. However, more pixels mean more visual tokens. Because LLM attention complexity grows quadratically with sequence length, doubling resolution can quadruple the computational cost, making it prohibitively expensive.

The authors aim to provide the LLM with multi-level visual features and high-resolution data without breaking the computational budget.

Methods

The authors introduce SparseCut, which consists of three primary innovations:

  • Sparse Shortcut Connections: Instead of one single bridge at the end, the authors create multiple "shortcuts" connecting intermediate ViT layers to intermediate LLM layers. They found that "sparse" connections (e.g., 8 shortcuts) actually perform better and are more stable than "dense" connections (connecting every layer), which can cause "information flooding" and training instability.
  • U-Shape Connection Order: Inspired by U-Net architectures in medical imaging, they found that connecting shallow visual layers (local details) to deep language layers (high-level reasoning) works best. This allows the model to use raw visual cues to ground its final complex decisions.
  • Efficient Multi-Grained Fusion: To handle high resolution, they don't just dump more tokens into the LLM. Instead, they use a cross-attention module within the shortcut. Low-resolution tokens act as "queries" to pool information from high-resolution "keys and values." This keeps the total number of tokens seen by the LLM constant, regardless of the input resolution.

Datasets

The models were trained using the standard LLaVA-1.5 paradigm:

  • Pretraining: 558K image-caption pairs.
  • Instruction Tuning: 665K multi-turn dialogues.

Evaluation was performed across a massive battery of benchmarks, including:

  • Academic Tasks: VQAv2, GQA, VizWiz, SciQA-IMG, and TextVQA.
  • Instruction Following/Hallucination: POPE, MMBench (English and Chinese), and SEEDBench.

Key Insights and Ideas

  • Sparsity is a Feature, Not a Bug: The "aha moment" here is that connecting every layer of the vision encoder to the LLM is counterproductive. Sparse connections act as a regularizer, forcing the model to pass only the most critical structural signals.
  • The U-Shape Advantage: The success of the U-shape suggests that the "semantic gap" is non-linear. High-level reasoning in the LLM benefits more from "raw" visual evidence than from the already-abstracted features of the final ViT layer.
  • Constant Context Length: By fusing resolutions before they hit the LLM, SparseCut achieves high-resolution performance (672x672) with the FLOPs of a low-resolution model. In their tests, SparseCut required only 9.6T FLOPs compared to 43.62T for a vanilla LLaVA model handling the same high-res input.

Key Results

  • Performance Gains: SparseCut-7B outperformed LLaVA-1.5 by an average of 2.2 percentage points across benchmarks. On the 13B scale, it showed a consistent 1.3 point improvement.
  • Hallucination Reduction: On the VizWiz benchmark (which includes "unanswerable" questions), SparseCut significantly reduced hallucinations. While LLaVA might guess an object in a blurry photo, SparseCut correctly identifies when the information is missing.
  • Efficiency: As noted, it achieves state-of-the-art results for its size while being nearly 4.5x more efficient than standard high-resolution scaling methods.
  • Stability: The architecture proved robust across different base models (Vicuna and Qwen) and remained stable even when unfreezing the vision encoder during training.

Limitations and Future Work

  • Comparison to Industrial Giants: While SparseCut beats models trained on similar data, it still lags behind "industrial-scale" models like Qwen2.5-VL, which are trained on trillions of tokens (compared to SparseCut's ~700 million).
  • Fixed Shortcut Patterns: The current shortcut patterns are manually designed (U-shape, uniform distribution). The authors suggest that future work could involve learned or dynamic shortcuts that adapt based on the complexity of the input image.
  • Modality Expansion: The paper focuses on vision-language, but the "Sparse Shortcut" concept could theoretically be applied to audio or video modalities, which the authors have not yet explored.

Summary updated

Abstract

With the remarkable success of large language models (LLMs) in natural language understanding and generation, multimodal large language models (MLLMs) have rapidly advanced in their ability to process data across multiple modalities. While most existing efforts focus on scaling up language models or constructing higher-quality training data, limited attention has been paid to effectively integrating cross-modal knowledge into the language space. In vision-language models, for instance, aligning modalities using only high-level visual features often discards the rich semantic information present in mid- and low-level features, limiting the model’s ability of cross-modality understanding. To address this issue, we propose SparseCut, a general cross-modal fusion architecture for MLLMs, introducing sparse shortcut connections between the cross-modal encoder and the LLM. These shortcut connections enable the efficient and hierarchical integration of visual features at multiple levels, facilitating richer semantic fusion without increasing computational overhead. We further introduce an efficient multi-grained feature fusion module, which performs the fusion of visual features before routing them through the shortcuts. This preserves the original language context and does not increase the overall input length, thereby avoiding an increase in computational complexity for the LLM. We systematically evaluate the performance of various shortcut patterns and demontrate that SparseCut can enhance the performance of MLLMs across various multimodal benchmarks with high training stability. It is also compatible with different base LLMs.
BibTeX
@misc{zhang2026sparse,
  author = {Jingrui Zhang and Yong Zhang and Feng Liang and Wei Wang and Runhao Zeng and Xiping Hu},
  title = {{Sparse Shortcuts: Facilitating Efficient Fusion in Multimodal Large Language Models}},
  year = {2026},
  eprint = {2602.00505},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2602.00505},
}
Figure