CoTA++: Understanding the Repeat Curse in dMLLMs from an Information Flow Perspective

Qiyan Zhao, Xiaofeng Zhang, Xu-Yao Zhang, Shijie Han, Zhepeng Li, Shunzhi Zhu, Da-Han WangarXiv 2026 · arXiv

TL;DR

Approximate caching makes diffusion-based multimodal LLMs (dMLLMs) fast, but it often triggers a "Repeat Curse" where models get stuck in stuttering loops. This paper diagnoses the curse as a breakdown in information flow and introduces CoTA++, a training-free, plug-and-play framework that eliminates these repetitions while maintaining high inference speeds.

What Problem Does This Paper Address?

Unlike traditional autoregressive LLMs that generate text one token at a time, diffusion-based multimodal LLMs (dMLLMs) decode by iteratively denoising an entire sequence in parallel. Because their attention is bidirectional—every token can "see" every other token—standard Key-Value (KV) caching doesn't work. To speed things up, researchers use "approximate caching," which reuses internal states across denoising steps.

However, this efficiency comes at a steep price: the Repeat Curse. As shown in the figure below, cached models frequently fall into loops, repeating a single token (like "the the the...") and breaking the sentence structure. This makes approximate caching—the current de facto standard for accelerating dMLLMs—highly unreliable for long-form generation.

Fig. 1: Motivation. LLaDA-V [7] responses to the same query: vanilla decoding is normal (top); dLLM-Cache [18] induces the Repeat Curse (repetitions in red, middle); CoTA++ removes them while retaining the acceleration (bottom).

Methods

The authors trace the Repeat Curse to three specific disruptions in how information flows through the model. They then propose CoTA++, which introduces a specific remedy for each disruption:

  1. Context-Token Attention Re-anchoring (CTAR): Caching causes attention to "drift" away from nearby context tokens (anchors). CTAR monitors which positions have missed context updates and re-calculates their attention routing to keep them "anchored."
  2. Decode-Aware Refresh (DAR): Standard caches refresh based on feature similarity, which often leaves the most critical positions (those about to be "committed" or unmasked) with stale data. DAR ensures that the cache budget is prioritized for tokens the model is currently deciding on.
  3. Context-Token Entropy-Guided Voting (CTEV): The authors found that when a model is about to repeat a token, the entropy (uncertainty) in its deep layers remains high. CTEV uses this entropy signal to "downvote" un-consolidated predictions, allowing more confident, non-repetitive candidates to take their place.
Fig. 3: Overview. Caching disrupts the information flow of context tokens on the three axes of dMLLM inference, and CoTA++ counteracts each disruption where it originates.

Datasets

The authors evaluated their method across three different dMLLMs (LLaDA-V, MMaDA, and LaViDa) and two caching backends (dLLM-Cache and SlowFast).

  • Repetition Analysis: Conducted on 500 MS-COCO image captions at various response lengths (64, 128, and 512 tokens).
  • General Benchmarks: Evaluated on nine standard multimodal benchmarks including DocVQA, ChartQA, MMStar, MME, SEED-Bench, MMBench, LLaVA-Bench, MathVista, and MathVerse.

Key Insights and Ideas

  • Anchors are Essential: In bidirectional models, local context tokens act as "anchors" for coherence. Caching breaks this by forcing queries to use outdated routing that doesn't account for newly unmasked neighbors.
  • Staleness vs. Similarity: A "fresh" cache isn't just about how much a feature has changed; it's about whether the data is timely for the specific tokens being decoded. Repetition concentrates at "stale" positions that the similarity-based refresh policy ignores.
  • Entropy as a Warning Light: A failure to "consolidate" information across layers (indicated by high deep-layer entropy) is a massive red flag for an impending repetition.
Fig. 2: The Repeat Curse and its three information-flow signatures. (a) Sample repetition rate on 500 MS-COCO captions, L=512. (b)–(d) LLaDA-V, L=128, vanilla against dLLM-Cache. (b) Within-step attention of suffix queries: caching disperses local anchoring (F1). (c) KV staleness at the decode moment along one response: most repeats (red) decode stale (F2). (d) Cross-layer entropy after generation at that response’s repeat positions and at normal positions: repeats fail to consolidate (F3).

Key Results

  • Repetition Nuked: For long responses (L=512), CoTA++ reduced the sample repetition rate from a staggering 99.6% down to 8.2% on LLaDA-V.
  • Accuracy Recovered: Caching typically causes a significant drop in benchmark scores. CoTA++ recovered most of this lost accuracy, bringing performance back to within 1-2 points of the "vanilla" (uncached) model.
  • Low Overhead: Despite the extra checks, CoTA++ adds less than 15% latency overhead compared to standard caching. It remains roughly 2.1x faster than decoding without any cache.

Limitations and Future Work

While CoTA++ is highly effective, the authors note a few areas for further exploration:

  • Hyperparameters: The method relies on a few fixed hyperparameters (like the penalty strength $\lambda$ and refresh slots $r$). Future work could involve making these budgets adaptive based on real-time model signals.
  • Model Scope: The study focused on current open-weight dMLLMs. As larger models (e.g., 100B+ parameters) emerge, the dynamics of the Repeat Curse and the optimal "re-anchoring" strategies may evolve.
  • Training Integration: Since CoTA++ is a decoding-time fix, a natural next step is investigating whether these information-flow insights can be used to train dMLLMs that are inherently more robust to caching.

Summary updated

Abstract

Diffusion-based multimodal large language models (dMLLMs) decode by iterative denoising with bidirectional attention, which precludes the standard autoregressive key--value cache. Practical deployments therefore rely on approximate caching that reuses intermediate states across denoising steps. Such caching, however, often makes a response repeat one token at adjacent positions in a run that breaks the sentence, a failure we term the \emph{Repeat Curse}. Tracing the context-token information flow reveals three disruptions. Within a step, caching disperses the attention that anchors a position on nearby context tokens. Across steps, similarity-based refresh is not staleness-aware, so some positions are decoded against stale key--value states. Across layers, the deep-layer entropy of the context fails to converge where repetition arises. We accordingly propose CoTA++, a training-free, plug-and-play framework that counteracts each disruption where it originates: Context-Token Attention Re-anchoring re-forms the attention routing against the current context, Decode-Aware Refresh reserves part of the cache's refresh budget for the positions about to be committed, and Context-Token Entropy-Guided Voting discounts the confidence of a position by the deep-layer entropy of its context, which reorders the unmasking. The three components act at non-overlapping stages of a decoding step and compose freely with existing dMLLMs and caches. Across three dMLLMs and two caching backends, CoTA++ reduces the sample repetition rate of long responses from up to 99.6\% to at most 17.0\%, recovers most of the accuracy lost to caching on nine benchmarks, and keeps the speedup of caching at under 15\% overhead, making approximate caching reliable for dMLLMs.
BibTeX
@misc{zhao2026cota,
  author = {Qiyan Zhao and Xiaofeng Zhang and Xu-Yao Zhang and Shijie Han and Zhepeng Li and Shunzhi Zhu and Da-Han Wang},
  title = {{CoTA++: Understanding the Repeat Curse in dMLLMs from an Information Flow Perspective}},
  year = {2026},
  eprint = {2610.23290},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.23290},
}
Figure