Abstract
Training Multimodal Large Language Models (MLLMs) is notoriously resource-intensive due to massive visual token sequences and deep network architectures. This paper introduces the Sparse Training Scheme (STS), a framework that accelerates training by compressing visual inputs and dynamically skipping model layers. By exploiting redundancies in both data and model parameters, STS cuts computational costs by over 15% while preserving up to 99.7% of the model's original performance.
Introduction
As language models evolve to handle multimodal inputs like images and text, their training costs have skyrocketed. A single image can introduce thousands of visual tokens—for instance, Qwen2.5-VL processes up to 16,384 visual tokens per image. This creates a massive memory and computational bottleneck during training. While research has focused on inference speed or text-only training efficiency, efficient training paradigms tailored for MLLMs remain largely underexplored. The Sparse Training Scheme (STS) addresses this gap by targeting two major sources of inefficiency: data redundancy in visual tokens and parameter redundancy in inter-layer computations.
Methodology: Sparsifying Data and Parameters
The STS framework splits the MLLM training pipeline into two stages, applying specific sparsification techniques to each:
1. Visual Token Compressor (VTC)
Applied during the modality alignment stage, VTC reduces the number of visual tokens fed into the LLM while the model weights are frozen. After testing random, instruction-guided, and uniform sampling, the authors found that Uniform Sampling—keeping tokens at fixed intervals—is most effective. This method discards redundant local details while preserving the global semantic structure necessary for visual understanding.
2. Layer Dynamic Skipper (LDS)
Applied during instruction fine-tuning, LDS dynamically skips transformer layers during both forward and backward passes. It utilizes two primary heuristics:
- Step-based decay: Training follows a "coarse-to-fine" trajectory. LDS skips more layers early in training when updates are coarse, gradually reducing the skip probability to zero as the model requires fine-grained tuning.
- Depth-based adjustment: Recognizing that shallow layers are critical for basic feature extraction, LDS assigns higher skip probabilities to deeper layers while keeping shallow layers active.
Experimental Results
The authors evaluated STS across five representative MLLMs, including LLaVA (Llama), Mipha (Phi-1.5 & Phi-2), and MoE-LLaVA (Qwen & StableLM), using benchmarks such as GQA, VQAv2, SQA, POPE, and MME.
Key results include:
- Compute Reduction: STS consistently reduced training FLOPs by over 15% (to approximately 82.2% - 83.9% of baseline levels).
- Performance Retention: Across all benchmarks, STS maintained 96.9% to 99.7% of baseline accuracy. The largest average performance drop was only 1.6 points.
- Task-Specific Impact: Robustness (POPE) and broad skills (MME) were largely unaffected, though fine-grained perception tasks (GQA, VQAv2) saw minor drops of 2-3 points.
Key Insights
- Uniformity beats cleverness: Surprisingly, simple uniform token dropping outperformed complex instruction-guided methods, which tended to bias the model away from basic visual perception like OCR.
- Training is "coarse-to-fine": The success of step-based decay proves that early training steps do not require the full depth of the network to converge effectively.
- Shallow layers are foundational: Depth-aware skipping reinforces that shallow layers do the heavy lifting for feature extraction, allowing deeper layers to be skipped more aggressively without catastrophic forgetting.
- Synergy is essential: While VTC or LDS alone may offer limited benefits or hurt accuracy, together they form a complementary system that balances efficiency and performance.
Conclusion
STS provides a practical and scalable solution for reducing the massive resource requirements of MLLM training. While aggressive sparsification causes minor trade-offs in fine-grained perception, the overall efficiency gains make it a compelling framework for future large-scale multimodal development. Future work may explore adaptive token compression to dynamically adjust rates based on visual complexity, potentially closing the remaining performance gap.
Summary updated