Stop-Think-AutoRegress: Language Modeling with Latent Diffusion Planning

Justin Lovelace, Christian Belardi, Sofian Zalouk, Adhitya Polavaram, Srivatsa Kundurthy, Kilian Q. Weinberger

TL;DR

The Stop-Think-AutoRegress Language Diffusion Model (STAR-LDM) introduces a "thinking" phase into text generation by combining autoregressive token prediction with latent diffusion planning. By pausing generation to iteratively refine a global semantic plan in a continuous embedding space, the model achieves superior narrative coherence, stronger commonsense reasoning, and highly flexible, plug-and-play attribute control without requiring model retraining.

What Problem Does This Paper Address?

Standard Large Language Models (LLMs) generate text autoregressively—predicting the next token sequentially from left to right. While highly successful, this token-by-token approach is inherently myopic. Unlike human writers, who pause, reflect, and outline a global plan before committing words to paper, standard autoregressive models make irrevocable local decisions at every step. This lack of lookahead makes long-range planning, structural consistency, and global coherence difficult to maintain.

Furthermore, steering the style, sentiment, or safety of an autoregressive model typically requires computationally expensive fine-tuning or reinforcement learning (like RLHF). While diffusion models excel at flexible, classifier-guided control, applying them directly to discrete text tokens remains a major challenge due to the non-differentiable nature of text.

Methods

STAR-LDM solves this by creating a unified hybrid architecture that operates across both discrete token spaces and continuous semantic spaces. The architecture consists of three core components:

  1. Autoregressive Decoder: A standard transformer backbone (implemented via a 770M parameter GPT2-Large) that ensures local grammatical fluency.
  2. Continuous Sentence Embedding Space: A latent space powered by Sentence-T5 XL (768 dimensions) that captures high-level semantic meaning while remaining invariant to shallow surface variations.
  3. Diffusion Transformers (DiTs): Two small transformer encoders that act as bridges, translating between the continuous embedding space and the discrete token representations used by the decoder.
Figure 1: Unified diffusion-guided language model architecture illustrating the training (left) and generation (right) processes. See text for details.

During inference, the model follows a three-step Stop-Think-AutoRegress workflow:

  • Stop: The model processes the user's input prefix text through the autoregressive decoder.
  • Think: Token generation pauses. Starting from pure Gaussian noise, the model performs iterative diffusion denoising (typically 50 steps) conditioned on the prefix to plan a coherent semantic embedding ($\tilde{z}_0$) for the upcoming text.
  • AutoRegress: The refined semantic plan is projected via a DiT into a sequence of soft prompt vectors. The autoregressive decoder then resumes, generating the final text continuation conditioned on both the original prefix and this new semantic plan.

The entire system is trained jointly using a multi-task objective that combines standard language modeling loss ($\mathcal{L}_{LM}$) with a denoising diffusion score-matching loss ($\mathcal{L}_{DM}$). For attribute steering (e.g., reducing toxicity), a lightweight MLP classifier is trained on noisy embeddings to guide the diffusion planning phase via plug-and-play classifier guidance.

Datasets

  • Pre-training: Trained on 16B tokens from the FineWeb dataset.
  • Evaluation Benchmarks: Evaluated on standard zero-shot Natural Language Understanding (NLU) tasks including CommonsenseQA (CSQA), Social IQA (SIQA), HellaSwag, Winogrande, PIQA, OpenBookQA, and ARC (Easy and Challenge).
  • Generation & Coherence: Evaluated using narrative continuations from the StoryCloze dataset and general text generation on the C4 validation set.
  • Attribute Control: The control classifiers were trained using the Jigsaw Unintended Bias dataset (for toxicity mitigation) and Amazon Polarity / Stanford Sentiment Treebank (SST-2) (for sentiment steering). Evaluation prompts were drawn from RealToxicityPrompts and OpenWebText.

Key Insights and Ideas

  • Semantic vs. Surface Guidance: Analysis of token log-likelihoods reveals that the latent diffusion plan specifically boosts the probability of content-bearing, narrative-critical words (e.g., specific nouns and key verbs) rather than simply altering surface-level syntax or grammar.
  • Continuous Planning Avoids Discrete Hurdles: Planning globally within a continuous sentence embedding space bypasses the optimization difficulties associated with discrete token diffusion, allowing the model to leverage the full mathematical maturity of continuous diffusion frameworks.
  • Tunable Noise Conditioning: Because the decoder is trained on varying noise levels, it learns to dynamically balance its inputs. At high noise levels, it relies more on the local prefix text; as the diffusion process reduces noise, it increasingly aligns its token choices with the emerging global semantic plan.

Key Results

  • Zero-Shot NLU Superiority: STAR-LDM (.96B total parameters) outperformed larger standard autoregressive models on language understanding benchmarks, achieving an average score of 47.6% compared to 43.6% for GPT2-Large (.77B) and 45.5% for GPT2-XL (1.5B). It showed particularly strong gains in commonsense reasoning tasks like CSQA.
  • Stronger Narrative Coherence: In blind head-to-head evaluations using Claude 3.7 Sonnet as a judge on the StoryCloze dataset, STAR-LDM achieved >70% win rates against GPT2-XL for narrative coherence and commonsense reasoning.
  • Better Control-Fluency Trade-offs: For plug-and-play attribute control, STAR-LDM achieved significantly better trade-offs between target attribute alignment and text fluency (measured by perplexity) compared to strong baselines like DeXperts, all while maintaining high lexical diversity.

Limitations and Future Work

The primary limitation of STAR-LDM is inference latency. Performing 15–50 iterative diffusion denoising steps during the "thinking" phase introduces significant computational overhead compared to pure autoregressive generation.

Furthermore, the authors note that their current implementation is not fully optimized; standard acceleration techniques like key-value (KV) caching are not yet integrated into the diffusion planning phase, meaning the prefix text is re-processed at every denoising step. Future work will focus on optimizing this inference pipeline and scaling the architecture to larger base models.

Summary updated

Abstract

The Stop-Think-AutoRegress Language Diffusion Model (STAR-LDM) integrates latent diffusion planning with autoregressive generation. Unlike conventional autoregressive language models limited to token-by-token decisions, STAR-LDM incorporates a "thinking" phase that pauses generation to refine a semantic plan through diffusion before continuing. This enables global planning in continuous space prior to committing to discrete tokens. Evaluations show STAR-LDM significantly outperforms similar-sized models on language understanding benchmarks and achieves $>70\%$ win rates in LLM-as-judge comparisons for narrative coherence and commonsense reasoning. The architecture also allows straightforward control through lightweight classifiers, enabling fine-grained steering of attributes without model retraining while maintaining better fluency-control trade-offs than specialized approaches.
BibTeX
@inproceedings{lovelace2025stop,
  author = {Justin Lovelace and Christian Belardi and Sofian Zalouk and Adhitya Polavaram and Srivatsa Kundurthy and Kilian Q. Weinberger},
  title = {{Stop-Think-AutoRegress: Language Modeling with Latent Diffusion Planning}},
  booktitle = {COLM},
  year = {2025},
  url = {https://openreview.net/forum?id=c05qIG1Z2B},
}
Figure