Self-Speculative Masked Diffusions

Andrew Campbell, Valentin De Bortoli, Jiaxin Shi, Arnaud Doucet

Masked Diffusion Models (MDMs) have emerged as powerful generative tools for discrete data, such as text and protein sequences. Unlike standard autoregressive models that generate tokens strictly from left to right, MDMs offer the flexibility to reveal tokens in any arbitrary order. However, this flexibility comes with a significant efficiency bottleneck.

When a standard MDM predicts the contents of masked positions, it relies on a "factorized" distribution—treating each masked position as independent. While mathematically convenient, this is a flawed assumption; in reality, token choices are highly interdependent. Consequently, revealing too many tokens simultaneously using this shortcut causes output quality to collapse. To maintain high fidelity, MDMs must take many tiny steps, resulting in a high Number of Function Evaluations (NFE) and sluggish generation speeds.

Self-Speculative Masked Diffusions (SSMD) introduces a novel approach that effectively doubles the generation speed of discrete diffusion models without sacrificing sample quality. By integrating a causal "verification" layer into a standard non-causal transformer architecture, SSMD enables a single-model version of speculative sampling that reveals multiple coherent tokens in a single forward pass.

A Hybrid Non-Causal/Causal Architecture

To overcome the factorization error, the authors designed a hybrid architecture that generates non-factorized predictions over masked positions in a single step.

Self-Speculative Masked Diffusion Architecture

The architecture is split into two primary components:

  1. The Draft (Non-Causal): The majority of the transformer blocks (e.g., 11 out of 12 layers) utilize standard "any-to-any" bidirectional attention. These layers act as a drafting mechanism, generating initial, factorized guesses for all masked positions simultaneously.
  2. The Target (Causal): The final layer is modified to employ a causal, permutation-informed attention mask. This layer acts as the "target" model. Because it enforces causality based on a specific generation ordering, it can properly account for the dependencies between the newly drafted tokens.

Instead of relying on two separate models (a small draft model and a large target model) as seen in traditional speculative decoding, SSMD uses the internal hidden states of its own non-causal blocks to feed the causal block. The target layer then validates the draft's guesses in parallel. Crucially, the architecture employs residual connections to pass hidden states from the non-causal layers directly to the causal output. This ensures the target distribution remains tightly aligned with the draft, significantly boosting the token acceptance rate during sampling.

Key Insights and Performance

The most significant breakthrough of SSMD is achieving a non-factorized (inter-dependent) predictive distribution in a single forward pass. Typically, obtaining such a distribution requires expensive autoregressive sampling. SSMD achieves this "for free" by re-masking the final layer, directly addressing the factorization error that limits MDM speed.

Remarkably, adding the causal layer and speculative logic introduces only 0.98% more FLOPs per forward pass compared to a standard transformer. The computational cost remains almost identical, but the utility is effectively doubled. Furthermore, because it uses a permutation-informed causal mask, the model retains the core advantage of MDMs: generating data in any order, which is vital for applications like protein design where a strict left-to-right direction does not exist.

The researchers evaluated SSMD across three distinct domains: Text8 (character-level spelling), OpenWebText (GPT-2 scale text modeling), and UniRef50 (protein sequence generation).

Key Results: * 2x Speedup: Across both text and protein tasks, SSMD achieved a ~2× reduction in NFE required to reach the same sample quality as standard MDMs. * Improved Text Quality: On the Text8 dataset, SSMD achieved significantly higher spelling accuracy at lower NFE ranges compared to baselines. On OpenWebText, it matched the generative perplexity of standard MDMs using half the network passes. * Better Protein Folding: For protein generation, SSMD produced sequences with higher pLDDT scores (a measure of folding confidence) at lower computational costs, indicating it captures underlying biological structures more effectively than factorized models.

Limitations and Future Directions

While the results at the 150M parameter scale are highly promising, it remains to be seen how this hybrid architecture behaves when scaled to multi-billion parameter Large Language Models (LLMs). Additionally, the authors found that an "11 non-causal + 1 causal" block split worked best for their scale. They hypothesize that adding more causal layers might weaken the draft model's performance by reducing its capacity, but further research is needed to identify the optimal split for different model sizes. Future work may also explore combining this method with advanced search-based inference techniques to push the boundaries of model reasoning even further.

Summary updated

Abstract

We present self-speculative masked diffusions, a new class of masked diffusion generative models for discrete data that require significantly fewer function evaluations to generate samples. Standard masked diffusion models predict factorized logits over currently masked positions. A number of masked positions are then sampled, however, the factorization approximation means that sampling too many positions in one go leads to poor sample quality. As a result, many simulation steps and therefore neural network function evaluations are required to generate high-quality data. We reduce the computational burden by generating \emph{non-factorized} predictions over masked positions. This is achieved by modifying the final transformer attention mask from non-causal to causal, enabling draft token generation and parallel validation via a novel, model-integrated speculative sampling mechanism. This results in a non-factorized predictive distribution over masked positions in a single forward pass. We apply our method to GPT2 scale text modelling and protein sequence generation, finding that we can achieve a ~2x reduction in the required number of network forward passes relative to standard masked diffusion models.
BibTeX
@inproceedings{campbell2026self,
  author = {Andrew Campbell and Valentin De Bortoli and Jiaxin Shi and Arnaud Doucet},
  title = {{Self-Speculative Masked Diffusions}},
  booktitle = {International Conference on Learning Representations},
  year = {2026},
  url = {https://openreview.net/forum?id=ogMTEtHO6M},
}
Figure