Sparse Training Scheme for Multimodal LLM

Kean Shi, Liang Chen, Haozhe Zhao, Baobao ChangarXiv 2025 · DOI · arXiv

Abstract

Training Multimodal Large Language Models (MLLMs) is notoriously resource-intensive due to massive visual token sequences and deep network architectures. This paper introduces the Sparse Training Scheme (STS), a framework that accelerates training by compressing visual inputs and dynamically skipping model layers. By exploiting redundancies in both data and model parameters, STS cuts computational costs by over 15% while preserving up to 99.7% of the model's original performance.

Introduction

As language models evolve to handle multimodal inputs like images and text, their training costs have skyrocketed. A single image can introduce thousands of visual tokens—for instance, Qwen2.5-VL processes up to 16,384 visual tokens per image. This creates a massive memory and computational bottleneck during training. While research has focused on inference speed or text-only training efficiency, efficient training paradigms tailored for MLLMs remain largely underexplored. The Sparse Training Scheme (STS) addresses this gap by targeting two major sources of inefficiency: data redundancy in visual tokens and parameter redundancy in inter-layer computations.

Methodology: Sparsifying Data and Parameters

The STS framework splits the MLLM training pipeline into two stages, applying specific sparsification techniques to each:

1. Visual Token Compressor (VTC)

Applied during the modality alignment stage, VTC reduces the number of visual tokens fed into the LLM while the model weights are frozen. After testing random, instruction-guided, and uniform sampling, the authors found that Uniform Sampling—keeping tokens at fixed intervals—is most effective. This method discards redundant local details while preserving the global semantic structure necessary for visual understanding.

2. Layer Dynamic Skipper (LDS)

Applied during instruction fine-tuning, LDS dynamically skips transformer layers during both forward and backward passes. It utilizes two primary heuristics:

  • Step-based decay: Training follows a "coarse-to-fine" trajectory. LDS skips more layers early in training when updates are coarse, gradually reducing the skip probability to zero as the model requires fine-grained tuning.
  • Depth-based adjustment: Recognizing that shallow layers are critical for basic feature extraction, LDS assigns higher skip probabilities to deeper layers while keeping shallow layers active.
The Sparse Training Scheme (STS) framework, featuring the Visual Token Compressor (VTC) and the Layer Dynamic Skipper (LDS).

Experimental Results

The authors evaluated STS across five representative MLLMs, including LLaVA (Llama), Mipha (Phi-1.5 & Phi-2), and MoE-LLaVA (Qwen & StableLM), using benchmarks such as GQA, VQAv2, SQA, POPE, and MME.

Comparison of STS performance and computational efficiency across multiple MLLM architectures.

Key results include:

  • Compute Reduction: STS consistently reduced training FLOPs by over 15% (to approximately 82.2% - 83.9% of baseline levels).
  • Performance Retention: Across all benchmarks, STS maintained 96.9% to 99.7% of baseline accuracy. The largest average performance drop was only 1.6 points.
  • Task-Specific Impact: Robustness (POPE) and broad skills (MME) were largely unaffected, though fine-grained perception tasks (GQA, VQAv2) saw minor drops of 2-3 points.

Key Insights

  • Uniformity beats cleverness: Surprisingly, simple uniform token dropping outperformed complex instruction-guided methods, which tended to bias the model away from basic visual perception like OCR.
  • Training is "coarse-to-fine": The success of step-based decay proves that early training steps do not require the full depth of the network to converge effectively.
  • Shallow layers are foundational: Depth-aware skipping reinforces that shallow layers do the heavy lifting for feature extraction, allowing deeper layers to be skipped more aggressively without catastrophic forgetting.
  • Synergy is essential: While VTC or LDS alone may offer limited benefits or hurt accuracy, together they form a complementary system that balances efficiency and performance.

Conclusion

STS provides a practical and scalable solution for reducing the massive resource requirements of MLLM training. While aggressive sparsification causes minor trade-offs in fine-grained perception, the overall efficiency gains make it a compelling framework for future large-scale multimodal development. Future work may explore adaptive token compression to dynamically adjust rates based on visual complexity, potentially closing the remaining performance gap.

Summary updated

Abstract

Multimodal Large Language Models (MLLMs) have demonstrated outstanding performance across a variety of domains. However, training MLLMs is often inefficient, as much of the computation is redundant due to the long input sequences from multimodal data and underutilized inter-layer operations. Notably, such redundancy is not static but varies across different stages of training. Building on this observation, we shift the focus to the training process itself and propose a training-efficient framework based on sparse representations, termed the Sparse Training Scheme (STS). Instead of applying a uniform sparsity strategy, STS adopts a stage-aware design that adapts to different sources of redundancy during training. Specifically, the framework consists of two complementary components: the Visual Token Compressor, which reduces the information load by compressing visual tokens during modality alignment, and the Layer Dynamic Skipper, which mitigates computational overhead by dynamically skipping unnecessary layers during instruction tuning. Our approach is broadly applicable to diverse MLLM architectures and has been extensively evaluated on multiple benchmarks, demonstrating its effectiveness and efficiency.
BibTeX
@misc{shi2025sparse,
  author = {Kean Shi and Liang Chen and Haozhe Zhao and Baobao Chang},
  title = {{Sparse Training Scheme for Multimodal LLM}},
  year = {2025},
  eprint = {2509.18150},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.18150},
}
Figure