Multi-Stream Residual Computation in Diffusion Language Models

Ekaterina Alimaskina, Suren Kazarian, Gleb Molodtsov, Aleksandr BeznosikovarXiv 2026 · arXiv

TL;DR

Diffusion language models (DLMs) perform two distinct tasks simultaneously: preserving visible context and refining masked predictions. This paper demonstrates that equipping Transformers with multiple residual streams allows the model to specialize, using one stream to "preserve" early context while another "evolves" predictions, leading to a ~5.7% reduction in reconstruction perplexity.

What Problem Does This Paper Address?

Standard Transformers process information through a single residual stream—a "highway" where every layer adds its output to the same running state. While this works for autoregressive models (where every token is essentially doing the same job: predicting the next one), it may be suboptimal for Diffusion Language Models.

In a DLM denoising pass, tokens have two very different roles:

  1. Visible tokens provide the ground-truth evidence needed for reconstruction.
  2. Masked tokens are blank slates that must gather information from the visible tokens to "fill in the blanks."

The authors argue that forcing these two complementary roles to share a single residual stream creates a bottleneck. They investigate whether providing multiple persistent streams and letting the model learn how to route information between them can improve performance and reveal how these models actually "think."

Methods

The researchers adapt the Hyper-Connections (mHC-lite) architecture for text diffusion. Instead of one residual stream, they use $n$ streams ($n=1$ to $4$).

Key technical components include:

  • Learned Routing: The model learns three token-conditioned mappings for each sublayer: what to read from the streams, how to mix existing streams, and where to write the new output.
  • Write Decomposition: They cleverly split the "write" operation into two parts: destination (which stream gets the update) and volume (how much the update changes the mean state).
  • Causal Interventions: To understand the routing, they perform "brain surgery" on the model—swapping routing patterns between masked and visible tokens or averaging them out to see what causes the most damage to performance.
  • Factorized Residual Routing (FRR): A simplified version of the architecture where routing is explicitly conditioned on whether a token is masked or visible, proving that this role distinction is the primary driver of the model's gains.

Datasets

  • Training: The models were pretrained on FineWeb-Edu (3.5 billion tokens).
  • Evaluation: To test generalization (transfer), they evaluated the models on four external corpora: OpenWebText, WikiText-103, LAMBADA, and C4.

Key Insights and Ideas

  • The "Prepare-Preserve-Reuse" Pattern: The model naturally organizes itself into three stages. Early layers prepare position-specific states. Then, one stream preserves this context (staying very similar to its early state) while the other stream evolves as it gathers evidence. Finally, late layers reuse that preserved context to finalize predictions.
Figure 1: Prepare–preserve–reuse within a denoising pass. Early blocks prepare position-specific states, one stream preserves them while the other accumulates evidence, and late blocks reuse the retained information. Arrows show selected read and write routes; weak connections are omitted.
  • The Mask is the Manager: Causal interventions revealed that the "visible vs. masked" status is the single most important factor for routing. Swapping the routing patterns between these roles is catastrophic for the model.
  • Two Streams are the Sweet Spot: While adding a second stream provides a massive boost, adding a third or fourth offers diminishing returns. Two streams capture over 86% of the potential gains from increasing stream count.
Figure 2: Perplexity reduction peaks early at two streams.
  • Volume vs. Destination: The destination of an update (which stream it goes to) is mostly determined by the token's role (masked/visible). However, the volume (how much to update) is highly specific to the individual token, especially for visible tokens in early layers and masked tokens in late layers.

Key Results

  • Perplexity Gains: The two-stream model reduced validation reconstruction perplexity by 5.71% compared to a standard Transformer baseline.
  • Robust Transfer: On average across four external datasets, the two-stream model achieved a 5.13% perplexity reduction, outperforming even the three- and four-stream variants in mean transfer.
  • Role-Conditioned Success: The simplified FRR model, which only knows if a token is masked or visible, recovered 50.2% of the gains seen in the fully learned two-stream model.
  • Scalability: These functional patterns (preservation and reuse) were successfully replicated in a larger 1-billion parameter model.

Limitations and Future Work

  • Scope: The study focuses specifically on masked diffusion. It remains to be seen if these multi-stream benefits translate to other diffusion formulations (like Gaussian or discrete state-space models).
  • Computational Cost: While the authors note the parameter increase is modest, multiple streams do increase memory usage and computation, which might be a concern for extremely long sequences.
  • Architectural Design: The authors suggest that rather than just adding streams to standard backbones, future work should explore designing architectures that are natively "role-aware" from the ground up.

Summary updated

Abstract

Diffusion language models reconstruct masked tokens in parallel, offering an alternative to autoregressive generation. Yet their Transformer backbones typically carry visible evidence and evolving masked predictions through a single residual stream. We equip masked diffusion models with multiple residual streams that persist through depth and adaptive routing between them, then investigate how they support these complementary roles. On the pretraining task, two streams reduce reconstruction perplexity by 5.71% on validation data and 5.13% on average across external corpora compared with standard residual connections. We observe that, without predefined roles, visible and masked positions learn distinct routing patterns. Causal interventions show that this distinction organizes read and write routing, while token-specific write strength remains important within each role. Guided by these findings, a model trained with role-conditioned routing recovers half the validation gain from adding a second stream. State interventions explain the benefit of the additional stream: it preserves early token representations for later reuse while evidence accumulates at masked positions. This mechanism recurs at a larger model scale, showing how persistent streams support context preservation alongside prediction refinement.
BibTeX
@misc{alimaskina2026multi,
  author = {Ekaterina Alimaskina and Suren Kazarian and Gleb Molodtsov and Aleksandr Beznosikov},
  title = {{Multi-Stream Residual Computation in Diffusion Language Models}},
  year = {2026},
  eprint = {2610.18407},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.18407},
}
Figure