TL;DR
Diffusion-based Multimodal Large Language Models (dMLLMs) often suffer from repetitive text generation when using caching for faster inference, a problem termed the "Repeat Curse." This paper reveals that context tokens act as crucial "anchors" for information flow, and caching disrupts this, leading to repetition. They propose CoTA, a novel method that enhances context token attention and penalizes uncertain predictions, effectively mitigating repetition.
What Problem Does This Paper Address?
Recent advancements in diffusion-based Multimodal Large Language Models (dMLLMs) have led to impressive capabilities, but their inference speed can be a bottleneck. To address this, researchers employ caching techniques to accelerate decoding. However, this optimization introduces a significant side effect: the generated text becomes excessively repetitive, a phenomenon the authors aptly name the "Repeat Curse." This repetition degrades the quality and readability of the model's outputs. The paper aims to understand the underlying mechanisms of this "Repeat Curse" in dMLLMs and propose an effective solution.
Methods
The core of this work lies in analyzing the "Repeat Curse" through the lens of information flow. The authors propose a novel method called CoTA (Context Tokens are Anchors), which has two key components:
- Context-Token Attention Enhancement (CTAE): This component aims to preserve the natural information flow by strengthening the attention paid to context tokens. It introduces a decay term that is applied element-wise to the attention matrix, prioritizing tokens that are semantically closer.
- Context-Token Entropy-Guided Voting (CTEV): This mechanism addresses the uncertainty that arises from repetitive outputs. It incorporates the aggregated deep-layer entropy of context tokens as a penalty term in the confidence score during decoding. This discourages the model from selecting uncertain tokens, which are often associated with repetition.
CoTA is designed as a plug-and-play method, meaning it can be integrated with existing dMLLMs and caching strategies without requiring any training.
Datasets
The paper evaluates its method on several multimodal benchmarks, including:
- MSCOCO (for image captioning)
- DocVQA
- ChartQA
- MMStar
- MME
- Seed
- LLaVA™
- MathVista
- MMBench
Key Insights and Ideas
- Context Tokens as Anchors: The paper establishes that in dMLLMs, context tokens (visual and instruction tokens) act as crucial "anchors" that aggregate semantic information across layers and guide the model's predictions.
- Information Flow Disruption: Caching, while speeding up inference, disrupts this natural information flow. It leads to a randomized attention distribution and weakens the stability of the model's output.
- Entropy as a Predictor of Repetition: The entropy of context tokens, which reflects the model's prediction certainty, normally converges in deeper layers. However, in the presence of repetition, this convergence fails, and entropy remains high, indicating model uncertainty.
- CoTA's Dual Approach: CoTA effectively tackles repetition by both preserving the intended information flow (CTAE) and penalizing uncertain predictions (CTEV), demonstrating a synergistic effect.
Key Results
- Significant Repetition Reduction: CoTA demonstrates significant effectiveness in alleviating repetition, reducing the adjacent repetition rate (ARR) by up to 92% in some cases. For instance, on the LLaDA-V model with caching, CoTA achieved an ARR of 1.2% compared to 13.9% for the cached baseline.
- Performance Improvement: CoTA consistently improves performance across various general multimodal tasks, showing gains in metrics like accuracy and ARR.
- Generalizability: The method is shown to be effective across different dMLLM architectures (e.g., LLaDA-V, MMaDA) and cache methods.
- Minimal Overhead: CoTA is a training-free method that incurs only modest computational overhead, with minimal impact on throughput and latency.
Limitations and Future Work
- Limited Scope of Validation: The authors acknowledge that CoTA has not yet been validated for generalizability across a wider range of open-source dMLLMs and base models of different scales due to limited research in this area.
- Scarcity of Cache Methods: The testing of CoTA is also limited by the scarcity of cache methods for dMLLMs, preventing its evaluation on a broader spectrum of caching approaches.
- Future Work: The authors suggest future work to address these limitations by exploring CoTA's applicability to a wider range of models and cache strategies.
Summary updated