Abstract Discrete diffusion-based multimodal large language models (dMLLMs) have emerged as a promising alternative to autoregressive MLLMs thanks to their advantages in parallel decoding and bidirectional context modeling. However, most existing dMLLMs incur significant computational overhead during inference due to the full-sequence attention computation in each denoising step. While pioneer studies attempt to resolve this issue via modality-agnostic optimizations, most overlook modality-specific visual token redundancy. In this work, we conduct a comprehensive study on how visual token redundancy evolves with different dMLLM architectures and tasks. Our study reveals that visual redundancy emerges primarily in from-scratch dMLLMs while handling long-answer tasks. Furthermore, we validate that from-scratch dMLLMs possess a unique ability to recover information lost during pruning progressively during late denoising steps. This work offers a new perspective on efficiency optimization for dMLLMs, advancing their applicability across various multimodal understanding tasks.
Introduction
Multimodal Large Language Models (MLLMs) have traditionally relied on autoregressive (AR) architectures, which generate tokens sequentially. Recently, discrete diffusion-based MLLMs (dMLLMs) have gained traction for their ability to perform parallel decoding and utilize bidirectional context. Despite these benefits, dMLLMs are computationally intensive because they require full-sequence attention across all text and visual tokens at every denoising step.
In the realm of AR models, visual token pruning is a common strategy to reduce redundancy and accelerate inference. This study investigates whether these established pruning techniques are applicable to dMLLMs, exploring the nuances of visual redundancy across different architectures and task types.
The "Visual-Hungry" Nature of dMLLMs
A key finding of this research is that dMLLMs are significantly more sensitive to visual information loss than their AR counterparts. Analysis of attention ratios reveals that dMLLMs maintain high attention toward visual tokens throughout the entire generation process. In contrast, AR models often exhibit a "visual-to-text" shift in later layers. This "visual-hungry" nature means that pruning strategies that work for AR models can often lead to catastrophic performance degradation in dMLLMs if not applied carefully.
The Restoration Ability of From-Scratch dMLLMs
One of the most significant discoveries in this study is the "restoration ability" inherent in from-scratch dMLLMs (e.g., LLaDA-V). Unlike AR-to-diffusion models, which are fine-tuned from pre-trained AR weights, from-scratch models can recover from the information loss caused by early-stage pruning.
Because these models utilize bidirectional attention, later denoising steps can effectively "fill in the blanks" left by pruned tokens. This recovery is most effective when pruning is initiated in the middle or late stages of the denoising process, rather than at the very beginning.
Architecture and Task-Specific Redundancy
The study highlights that the effectiveness of pruning is highly dependent on both the model architecture and the nature of the task:
- Architecture Matters: AR-to-diffusion models (like LaViDa-Dream) lack the restoration ability of from-scratch models. They retain an "autoregressive tendency" that makes them brittle; once information is pruned, the loss is often permanent.
- Task Sensitivity: Redundancy is not uniform across tasks. Short-answer tasks (e.g., POPE, MME) require high visual fidelity, and even moderate pruning can lead to significant accuracy drops. Conversely, long-answer tasks (e.g., detailed video or document description) exhibit substantial redundancy that can be exploited for speedups.
Experimental Results and Efficiency Gains
By applying strategic pruning (such as DivPrune) during the middle decoding steps of from-scratch models, the researchers achieved a 1.44x speedup on long-answer benchmarks like InfoVQA and DocVQA with negligible loss in accuracy. However, the same techniques applied to short-answer tasks often resulted in performance degradation, suggesting that "easy" tasks in the diffusion framework actually necessitate higher visual token retention.
Conclusion
This comprehensive study provides a roadmap for optimizing dMLLM inference. While standard pruning techniques from the AR domain are not directly transferable, the unique restoration ability of from-scratch dMLLMs opens new doors for adaptive, step-aware pruning. Future work should focus on developing dynamic pruning schedules that account for the specific denoising dynamics of these models, while exploring layer-skipping as a more robust alternative for AR-adapted diffusion architectures.
Summary updated