Abstract
Masked Diffusion Models (MDMs) provide a flexible alternative to autoregressive models for discrete data generation, such as text and protein sequences. However, MDMs typically suffer from slow inference speeds due to the "factorization problem," where independent token predictions require numerous refinement steps to maintain quality. This paper, presented for ICLR 2026, introduces "Self-Speculative Masked Diffusions," a novel architecture that blends non-causal and causal transformer blocks into a single model. By using the non-causal layers to draft tokens and a final causal layer to verify them in parallel, the proposed method achieves a ~2x speedup in generation across multiple benchmarks, including text and protein modeling, with negligible computational overhead.
Introduction
In the rapidly evolving landscape of generative AI, a persistent trade-off exists between the precision of Autoregressive (AR) models and the speed of Masked Diffusion Models (MDMs). AR models, such as the GPT family, generate sequences by predicting one token at a time from left to right. While this sequential nature ensures high quality by capturing complex dependencies, it is inherently slow for long sequences. MDMs offer a more flexible alternative, starting with a fully masked sequence and iteratively unmasking tokens. This approach is particularly valuable for non-linear data like protein sequences, where a fixed generation order is not always optimal.
However, MDMs face a significant hurdle known as the "factorization problem." During each generation step, the model assumes that all currently masked tokens are independent of one another. In reality, tokens are highly interdependent; attempting to reveal too many at once based on independent probabilities often results in incoherent outputs. To maintain quality, MDMs must perform numerous small steps, each requiring a full forward pass through the network, which negates their theoretical speed advantage. This paper proposes a hybrid architecture that overcomes this limitation by integrating speculative sampling directly into the model's design.
Methods: The Self-Speculative Architecture
The core innovation of this work is a hybrid architecture that enables speculative sampling within a single model. Unlike traditional speculative decoding that requires separate "drafter" and "verifier" models, this approach integrates both functions into a unified transformer stack.
Hybrid Transformer Design
The model consists of a stack of transformer blocks where the majority are non-causal, followed by a small number of causal blocks (specifically $\sigma$-GPT blocks). The non-causal layers serve as the "drafter," proposing values for all masked positions simultaneously. The causal layers then act as the "verifier," calculating the conditional probability of tokens given previously accepted ones in a single forward pass.
Integration and Residual Connections
A key technical detail is the use of residual connections to add non-causal hidden states to the causal outputs. This allows the causal "target" model to focus on learning the dependencies that the non-causal "draft" model might have missed, effectively aligning the two distributions and boosting the token acceptance rate. This integration ensures that the verification step is not just a filter, but a constructive refinement of the draft.
Flexible Sampling
The method supports flexible unmasking orders ($\sigma$), which is particularly beneficial for biological data like protein sequences that lack a natural left-to-right progression. The sampling algorithm formalizes how the model draws generation orders and reveals tokens, allowing for efficient parallel verification regardless of the sequence's structure.
Experiments and Results
The researchers evaluated the self-speculative approach on three benchmarks: Text8 (character-level text), OpenWebText (GPT-2 scale text), and UniRef50 (protein sequences).
Performance and Efficiency
Across all tasks, the method achieved a consistent ~2x speedup. On OpenWebText (150M parameters), the model maintained generative perplexity and sample diversity identical to standard MDM baselines while requiring half the Network Forward Evaluations (NFEs). In protein generation tasks using UniRef50, the model produced sequences with higher pLDDT scores (indicating better folding potential) at lower computational budgets compared to standard MDMs. This suggests that the causal verification layer is particularly adept at capturing the structural dependencies crucial for protein design.
Computational Overhead
The addition of the causal verification layer is highly efficient. For a 12-layer model (11 non-causal, 1 causal), the increase in FLOPs for a single forward pass is approximately 0.98%. This marginal cost is far outweighed by the 50% reduction in the total number of passes required for high-quality generation. Furthermore, the method outperformed "Self-Distillation Through Time" (SDTT) by maintaining higher sample diversity, avoiding the "mode-seeking" behavior often found in distilled models.
Conclusion
Self-Speculative Masked Diffusions address the primary bottleneck of discrete diffusion models by internalizing the speculative decoding process. By leveraging a hybrid architecture with residual connections, the model effectively bridges the gap between independent drafting and dependent verification. This work demonstrates that self-speculation is a powerful tool for accelerating generative models without sacrificing quality, offering a promising path forward for efficient foundation models in both language and biology.
Summary updated