A Comprehensive Study on Visual Token Redundancy for Discrete Diffusion-based Multimodal Large Language Models

Duo Li, Zuhao Yang, Xiaoqin Zhang, Ling Shao, Shijian Lu

Abstract Discrete diffusion-based multimodal large language models (dMLLMs) have emerged as a promising alternative to autoregressive MLLMs thanks to their advantages in parallel decoding and bidirectional context modeling. However, most existing dMLLMs incur significant computational overhead during inference due to the full-sequence attention computation in each denoising step. While pioneer studies attempt to resolve this issue via modality-agnostic optimizations, most overlook modality-specific visual token redundancy. In this work, we conduct a comprehensive study on how visual token redundancy evolves with different dMLLM architectures and tasks. Our study reveals that visual redundancy emerges primarily in from-scratch dMLLMs while handling long-answer tasks. Furthermore, we validate that from-scratch dMLLMs possess a unique ability to recover information lost during pruning progressively during late denoising steps. This work offers a new perspective on efficiency optimization for dMLLMs, advancing their applicability across various multimodal understanding tasks.

Introduction

Multimodal Large Language Models (MLLMs) have traditionally relied on autoregressive (AR) architectures, which generate tokens sequentially. Recently, discrete diffusion-based MLLMs (dMLLMs) have gained traction for their ability to perform parallel decoding and utilize bidirectional context. Despite these benefits, dMLLMs are computationally intensive because they require full-sequence attention across all text and visual tokens at every denoising step.

In the realm of AR models, visual token pruning is a common strategy to reduce redundancy and accelerate inference. This study investigates whether these established pruning techniques are applicable to dMLLMs, exploring the nuances of visual redundancy across different architectures and task types.

The "Visual-Hungry" Nature of dMLLMs

A key finding of this research is that dMLLMs are significantly more sensitive to visual information loss than their AR counterparts. Analysis of attention ratios reveals that dMLLMs maintain high attention toward visual tokens throughout the entire generation process. In contrast, AR models often exhibit a "visual-to-text" shift in later layers. This "visual-hungry" nature means that pruning strategies that work for AR models can often lead to catastrophic performance degradation in dMLLMs if not applied carefully.

The Restoration Ability of From-Scratch dMLLMs

One of the most significant discoveries in this study is the "restoration ability" inherent in from-scratch dMLLMs (e.g., LLaDA-V). Unlike AR-to-diffusion models, which are fine-tuned from pre-trained AR weights, from-scratch models can recover from the information loss caused by early-stage pruning.

Because these models utilize bidirectional attention, later denoising steps can effectively "fill in the blanks" left by pruned tokens. This recovery is most effective when pruning is initiated in the middle or late stages of the denoising process, rather than at the very beginning.

Figure 1: Accuracy of dMLLMs as a function of retention ratios (left) and the starting denoising step for pruning (right). The right plot highlights the restoration ability, where accuracy improves as pruning is delayed to later steps.

Architecture and Task-Specific Redundancy

The study highlights that the effectiveness of pruning is highly dependent on both the model architecture and the nature of the task:

  1. Architecture Matters: AR-to-diffusion models (like LaViDa-Dream) lack the restoration ability of from-scratch models. They retain an "autoregressive tendency" that makes them brittle; once information is pruned, the loss is often permanent.
  2. Task Sensitivity: Redundancy is not uniform across tasks. Short-answer tasks (e.g., POPE, MME) require high visual fidelity, and even moderate pruning can lead to significant accuracy drops. Conversely, long-answer tasks (e.g., detailed video or document description) exhibit substantial redundancy that can be exploited for speedups.
Figure 5: Performance comparison across various benchmarks. The data shows that from-scratch dMLLMs (top) maintain better performance under pruning compared to AR-to-diffusion models (bottom), especially in long-answer tasks.

Experimental Results and Efficiency Gains

By applying strategic pruning (such as DivPrune) during the middle decoding steps of from-scratch models, the researchers achieved a 1.44x speedup on long-answer benchmarks like InfoVQA and DocVQA with negligible loss in accuracy. However, the same techniques applied to short-answer tasks often resulted in performance degradation, suggesting that "easy" tasks in the diffusion framework actually necessitate higher visual token retention.

Conclusion

This comprehensive study provides a roadmap for optimizing dMLLM inference. While standard pruning techniques from the AR domain are not directly transferable, the unique restoration ability of from-scratch dMLLMs opens new doors for adaptive, step-aware pruning. Future work should focus on developing dynamic pruning schedules that account for the specific denoising dynamics of these models, while exploring layer-skipping as a more robust alternative for AR-adapted diffusion architectures.

Summary updated

Abstract

Discrete diffusion-based multimodal large language models (dMLLMs) have emerged as a promising alternative to autoregressive MLLMs thanks to their advantages in parallel decoding and bidirectional context modeling, but most existing dMLLMs incur significant computational overhead during inference due to the full-sequence attention computation in each denoising step. Pioneer studies attempt to resolve this issue from a modality-agnostic perspective via key-value cache optimization or efficient sampling but most of them overlook modality-specific visual token redundancy. In this work, we conduct a comprehensive study on how visual token redundancy evolves with different dMLLM architectures and tasks and how visual token pruning affects dMLLM responses and efficiency. Specifically, our study reveals that visual redundancy emerges only in from-scratch dMLLMs while handling long-answer tasks. In addition, we validate that visual token pruning introduces non-negligible information loss in dMLLMs and only from-scratch dMLLMs can recover the lost information progressively during late denoising steps. Furthermore, our study shows that layer-skipping is promising for accelerating AR-to-diffusion dMLLMs, whereas progressive or late-step pruning is more effective for from-scratch dMLLMs. Overall, this work offers a new perspective on efficiency optimization for dMLLMs, greatly advancing their applicability across various multimodal understanding tasks.
BibTeX
@misc{li2025comprehensive,
  author = {Duo Li and Zuhao Yang and Xiaoqin Zhang and Ling Shao and Shijian Lu},
  title = {{A Comprehensive Study on Visual Token Redundancy for Discrete Diffusion-based Multimodal Large Language Models}},
  year = {2025},
  eprint = {2511.15098},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2511.15098},
}
Figure