DRAMA: Dissecting Attention Redundancy for Accelerating Multimodal Diffusion Large Language Models

Yi-Hsin Hung, Fangfu Liu, Shao-Yuan Lo, Chu-Song ChenarXiv 2026 · arXiv

TL;DR

DRAMA is a training-free framework that accelerates multimodal diffusion Large Language Models (dLLMs) by up to 18.67×. By systematically measuring attention redundancy, the authors developed two plug-and-play mechanisms—visual token pruning and adaptive stopping—that bring these models from "impractically slow" to "interactive speeds" without sacrificing accuracy.

What Problem Does This Paper Address?

While autoregressive models (like GPT-4) generate text token-by-token, diffusion LLMs (dLLMs) generate text by iteratively denoising a fully masked sequence. This allows for parallel generation and global context reasoning, but it comes with a "quadratic tax."

In multimodal settings, the problem is even worse. A single image can be decomposed into 2,000–3,000 visual tokens. Because dLLMs use bidirectional attention, every single denoising step requires a full forward pass over the entire sequence. Since visual tokens dominate the sequence length and remain static throughout the process, the model repeatedly spends massive computational effort re-processing the same visual information.

Previous attempts to speed this up often relied on "vibe-based" heuristics or isolated visualizations. The authors of DRAMA sought a measurement-grounded approach to find exactly where the redundancy lies and how to cut it out.

Methods

The researchers first performed a "two-axis dissection" of attention redundancy:

  1. Spatial Axis: They found that attention mass is extremely concentrated. A tiny fraction of visual tokens (the "salient" parts of an image) does almost all the heavy lifting.
  2. Temporal Axis: They discovered that after an initial "formation phase," the attention distributions stabilize into a "plateau" where subsequent denoising steps add very little new information.

Based on these insights, they proposed DRAMA (Dissecting attention Redundancy for Accelerating MultimodAl dLLMs), which consists of two core components:

  • Spatial-Redundancy-Aware Visual Token Pruning (SVTP): Before denoising even starts, the model scores image patches based on their relevance to the input question. It keeps only the most relevant patches, drastically shortening the sequence length for all subsequent steps with zero per-step overhead.
  • Sufficiency-Guided Adaptive Stopping (SGAS): Instead of running for a fixed number of steps (e.g., 64), the model monitors the decoded text. Once the output reaches "semantic sufficiency" (it answers the question) and "lexical coherence" (it isn't gibberish or repetitive), the process terminates early.
Overview of the DRAMA framework showing SVTP and SGAS mechanisms.

Datasets

The framework was evaluated across six diverse vision-language benchmarks:

  • AI2D: Science diagrams.
  • MMBench & MM-Star: General multimodal reasoning.
  • MME: Perceptual and cognitive tasks.
  • MMMU: Massive multi-discipline understanding.
  • SEED-Bench: Generative comprehension.

They also tested generalization on POPE (object hallucination) and TextVQA (text-heavy images).

Key Insights and Ideas

  • The 10% Rule: On average, the top 10% of visual key tokens capture roughly 78% of a query's attention mass. This confirms that most visual tokens are "near-irrelevant context" for a given question.
  • The Three-Phase Denoising Trajectory: Denoising isn't a linear progression. It starts with Structure Formation (high information gain), enters a long Stable Plateau (high redundancy), and ends with Terminal Refinement (fine-grained disambiguation). DRAMA targets that middle plateau for maximum savings.
  • Attention Dilution: Interestingly, pruning visual tokens sometimes improves accuracy (as seen in the MME benchmark). This suggests that too much visual data can actually "dilute" the model's focus, and pruning acts as a helpful filter.
Figure 2: Spatial redundancy analysis. (a) Per-sample attention share of each image patch ranked by dominance. (b) Per-sample attention share distribution of the most-attended patch across the dataset. (c) Cumulative attention coverage by top-k% visual key tokens, stratified by query-key modality pair. (d) Minimum visual tokens needed for 90% attention coverage (κ(0.9)) across denoising steps

Key Results

DRAMA achieves a state-of-the-art Pareto frontier for dLLM acceleration:

  • Massive Speedup: It reached an average 18.67× speedup (and up to 30× on specific tasks like AI2D) compared to the 64-step baseline.
  • Interactive Throughput: It boosted throughput from a sluggish 2.6 tokens/s to 51.9 tokens/s, making dLLMs viable for real-time applications.
  • Accuracy Retention: Despite the aggressive pruning and early stopping, DRAMA maintained competitive accuracy, often outperforming fixed-budget baselines that used significantly more compute.
Figure 1: Accuracy-latency Pareto frontier. DRAMA variants (diamonds) dominate the Pareto front, achieving higher accuracy at lower latency than all fixed-budget baselines (triangles).

Limitations and Future Work

  • Task Sensitivity: DRAMA is least effective on "text-dense" images (like those in TextVQA). When an image contains a lot of small, relevant text, pruning visual tokens too aggressively can lead to information loss.
  • No New Capabilities: As a training-free acceleration method, DRAMA makes models faster but does not improve their underlying reasoning or "intelligence."
  • Future Directions: The authors suggest that future work could explore more dynamic pruning strategies or integrate these insights into the training phase of dLLMs to create inherently more efficient architectures.

Summary updated

Abstract

Multimodal diffusion large language models (dLLMs) generate text through iterative masked denoising with bidirectional attention, offering a compelling alternative to autoregressive vision-language modeling. However, each denoising step requires a full forward pass over long multimodal sequences in which visual tokens dominate the sequence length, and a fixed step budget further compounds this quadratic per-step cost. Existing inference acceleration approaches for multimodal dLLMs characterize attention redundancy implicitly, motivating designs from isolated attention visualizations rather than from measurement. We close this gap with DRAMA, a training-free acceleration framework grounded in a systematic dissection of attention redundancy in multimodal dLLMs. By probing attention tensors along spatial and temporal axes, we reveal two persistent redundancy structures: (i) attention mass consistently concentrates on a sparse subset of visual tokens, and (ii) attention distributions rapidly stabilize across denoising steps after an initial formation phase. These findings motivate two complementary mechanisms: Spatial-Redundancy-Aware Visual Token Pruning (SVTP) and Sufficiency-Guided Adaptive Stopping (SGAS). Together, DRAMA achieves an average 18.67 times inference speedup across six vision-language benchmarks without training, architectural modification, or attention caching, while maintaining competitive accuracy. Our results establish a strong, measurement-grounded acceleration baseline for multimodal dLLM inference.
BibTeX
@misc{hung2026drama,
  author = {Yi-Hsin Hung and Fangfu Liu and Shao-Yuan Lo and Chu-Song Chen},
  title = {{DRAMA: Dissecting Attention Redundancy for Accelerating Multimodal Diffusion Large Language Models}},
  year = {2026},
  eprint = {2611.16345},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.16345},
}
Figure