Set Diffusion: Interpolating Token Orderings between Autoregression and Diffusion for Fast and Flexible Decoding

Marianne Arriola, Volodymyr Kuleshov

Introduction

In the landscape of Large Language Models (LLMs), researchers typically face a strict tradeoff between flexibility and speed. Autoregressive (AR) models, such as GPT-4, generate text sequentially one token at a time. While this allows for Key-Value (KV) caching—storing previous computations to save time—the sequential nature fundamentally bottlenecks generation speed. Conversely, discrete diffusion models can generate multiple tokens in parallel, offering significant speedups. However, standard diffusion models are constrained to fixed-length outputs, lack KV caching support due to their bidirectional context requirements, and struggle with variable-length tasks.

Prior attempts to find a middle ground, such as "Block Diffusion," generated text in sequential blocks. Yet, these models remained restricted by rigid left-to-right structures and could only update their KV cache after an entire block was completed. To resolve these limitations, researchers have introduced Set Diffusion, a novel framework that factorizes text generation into flexible-position "token sets" rather than fixed blocks, enabling variable-length generation, efficient KV caching, and high-performance infilling.

The Set Diffusion Framework

Set Diffusion treats a sequence as a series of arbitrary sets of tokens, rather than rigid blocks. This conceptual shift is powered by three core innovations:

1. Set-Causal Architecture The authors designed a transformer utilizing a "set-causal" attention mask. This allows the model to attend to all previously "revealed" token sets. Crucially, this architecture supports KV cache updates after every single inference step—mirroring the efficiency of AR models but applying it to multiple tokens simultaneously. By enabling KV caching in a diffusion framework, Set Diffusion eliminates the re-computation penalty that typically makes diffusion models prohibitively expensive for long sequences.

2. Position-Offset Schedules To control the generation process, the model employs a "reveal-time" schedule. By adjusting a window size parameter, users can smoothly interpolate between pure autoregression (generating one token at a time) and pure diffusion (generating all tokens at once). This flexibility allows practitioners to tune the model dynamically for either maximum output quality or maximum parallel speed.

3. Sliding-Window SetDLMs The researchers propose Sliding-Window Set Diffusion Language Models (SW-SetDLMs). During training, these models learn to predict tokens in a randomized order as "singleton sets." This approach ensures the model sees every token as both context and target in a single forward pass, significantly reducing gradient variance and accelerating convergence compared to standard masked diffusion. At inference time, a sliding window decodes multiple tokens in parallel while maintaining a left-to-right bias to preserve text quality.

Experimental Results

Set Diffusion was evaluated across a diverse suite of tasks, including mathematical reasoning (GSM8K), summarization (CNN/DailyMail), infilling (ROCStories), and language modeling (OpenWebText and LM1B).

  • Speed-Quality Pareto Frontier: On the GSM8K math benchmark, Set Diffusion achieved a strictly better tradeoff between speed and accuracy than Block Diffusion. It reached higher accuracy while generating text faster than its block-based counterparts.
  • Superior Infilling: Because Set Diffusion is not locked into a strict left-to-right generation order, it naturally excels at infilling tasks. On ROCStories, it substantially outperformed Block Diffusion, achieving higher ROUGE scores with approximately 20% faster decoding.
  • State-of-the-Art Perplexity: On the LM1B dataset, Set Diffusion achieved the best test perplexity among all evaluated diffusion models, even surpassing some AR baselines.

Limitations and Future Directions

While Set Diffusion represents a significant leap forward for non-autoregressive language modeling, a few challenges remain. First, the very best AR models still maintain a slight edge in pure sample quality over hybrid diffusion models at equivalent scales. Second, the position-offset schedule currently relies on manual hyperparameter tuning; future work could explore learning the optimal generation order automatically during training. Finally, to fully realize the theoretical speedups of the set-causal mask, custom-optimized GPU kernels—similar to FlashAttention—must be developed to handle these non-standard masking patterns efficiently.

Summary updated

Abstract

Discrete diffusion models have steadily improved in quality relative to autoregressive (AR) models. However, these models are normally constrained to fixed-length generation and do not support key-value (KV) caching. Block diffusion partially bridges diffusion and AR by unmasking token blocks left-to-right, but it is still limited to generate fixed-size blocks sequentially. Here, we present a new class of language models, set diffusion, comprised of (i) a likelihood parameterization that factorizes over flexible-position, flexible-length token sets and (ii) a set-causal diffusion architecture that supports KV cache updates after every inference step. By factorizing over token sets instead of fixed-size blocks, tokens can be decoded in arbitrarily-ordered sets, including sliding-window sets, enabling faster inference and support for any-order decoding. Set diffusion achieves better speed-quality tradeoffs on mathematical reasoning, summarization, and unconditional generation compared to prior diffusion language models while offering stronger infilling performance than block diffusion. We provide the code, along with the model weights and blog post on the project page: [https://m-arriola.com/setdlms/](https://m-arriola.com/setdlms/)
BibTeX
@inproceedings{arriola2026set,
  author = {Marianne Arriola and Volodymyr Kuleshov},
  title = {{Set Diffusion: Interpolating Token Orderings between Autoregression and Diffusion for Fast and Flexible Decoding}},
  booktitle = {International Conference on Machine Learning},
  year = {2026},
  url = {https://openreview.net/forum?id=tN8f0dxQ9i},
}
Figure