TL;DR
Approximate caching makes diffusion-based multimodal LLMs (dMLLMs) fast, but it often triggers a "Repeat Curse" where models get stuck in stuttering loops. This paper diagnoses the curse as a breakdown in information flow and introduces CoTA++, a training-free, plug-and-play framework that eliminates these repetitions while maintaining high inference speeds.
What Problem Does This Paper Address?
Unlike traditional autoregressive LLMs that generate text one token at a time, diffusion-based multimodal LLMs (dMLLMs) decode by iteratively denoising an entire sequence in parallel. Because their attention is bidirectional—every token can "see" every other token—standard Key-Value (KV) caching doesn't work. To speed things up, researchers use "approximate caching," which reuses internal states across denoising steps.
However, this efficiency comes at a steep price: the Repeat Curse. As shown in the figure below, cached models frequently fall into loops, repeating a single token (like "the the the...") and breaking the sentence structure. This makes approximate caching—the current de facto standard for accelerating dMLLMs—highly unreliable for long-form generation.
Methods
The authors trace the Repeat Curse to three specific disruptions in how information flows through the model. They then propose CoTA++, which introduces a specific remedy for each disruption:
- Context-Token Attention Re-anchoring (CTAR): Caching causes attention to "drift" away from nearby context tokens (anchors). CTAR monitors which positions have missed context updates and re-calculates their attention routing to keep them "anchored."
- Decode-Aware Refresh (DAR): Standard caches refresh based on feature similarity, which often leaves the most critical positions (those about to be "committed" or unmasked) with stale data. DAR ensures that the cache budget is prioritized for tokens the model is currently deciding on.
- Context-Token Entropy-Guided Voting (CTEV): The authors found that when a model is about to repeat a token, the entropy (uncertainty) in its deep layers remains high. CTEV uses this entropy signal to "downvote" un-consolidated predictions, allowing more confident, non-repetitive candidates to take their place.
Datasets
The authors evaluated their method across three different dMLLMs (LLaDA-V, MMaDA, and LaViDa) and two caching backends (dLLM-Cache and SlowFast).
- Repetition Analysis: Conducted on 500 MS-COCO image captions at various response lengths (64, 128, and 512 tokens).
- General Benchmarks: Evaluated on nine standard multimodal benchmarks including DocVQA, ChartQA, MMStar, MME, SEED-Bench, MMBench, LLaVA-Bench, MathVista, and MathVerse.
Key Insights and Ideas
- Anchors are Essential: In bidirectional models, local context tokens act as "anchors" for coherence. Caching breaks this by forcing queries to use outdated routing that doesn't account for newly unmasked neighbors.
- Staleness vs. Similarity: A "fresh" cache isn't just about how much a feature has changed; it's about whether the data is timely for the specific tokens being decoded. Repetition concentrates at "stale" positions that the similarity-based refresh policy ignores.
- Entropy as a Warning Light: A failure to "consolidate" information across layers (indicated by high deep-layer entropy) is a massive red flag for an impending repetition.
Key Results
- Repetition Nuked: For long responses (L=512), CoTA++ reduced the sample repetition rate from a staggering 99.6% down to 8.2% on LLaDA-V.
- Accuracy Recovered: Caching typically causes a significant drop in benchmark scores. CoTA++ recovered most of this lost accuracy, bringing performance back to within 1-2 points of the "vanilla" (uncached) model.
- Low Overhead: Despite the extra checks, CoTA++ adds less than 15% latency overhead compared to standard caching. It remains roughly 2.1x faster than decoding without any cache.
Limitations and Future Work
While CoTA++ is highly effective, the authors note a few areas for further exploration:
- Hyperparameters: The method relies on a few fixed hyperparameters (like the penalty strength $\lambda$ and refresh slots $r$). Future work could involve making these budgets adaptive based on real-time model signals.
- Model Scope: The study focused on current open-weight dMLLMs. As larger models (e.g., 100B+ parameters) emerge, the dynamics of the Repeat Curse and the optimal "re-anchoring" strategies may evolve.
- Training Integration: Since CoTA++ is a decoding-time fix, a natural next step is investigating whether these information-flow insights can be used to train dMLLMs that are inherently more robust to caching.
Summary updated