HiLo-RD: Cross-Budget Self-Distillation for Masked Diffusion Language Models

Xiang Li, Chengting Yu, Jungang Li, Yilun Zheng, Junzhuo Li, Xingyao Zhang, Wenbo Su, Xuming HuarXiv 2026 · arXiv

TL;DR

HiLo-RD (High-to-Low Budget Representation Distillation) is a novel self-distillation method that significantly improves the reasoning capabilities of Masked Diffusion Language Models (MDLMs) under tight generation budgets. By teaching the model to match its own internal representations from a larger generation window, it avoids "prematurely committing" to answers and produces responses that are both more accurate and more concise.

What Problem Does This Paper Address?

Unlike traditional autoregressive models that generate text strictly from left to right, Masked Diffusion Language Models (MDLMs) can generate tokens in any order and decide when to "commit" (fix) a token. This flexibility is a double-edged sword. When operating under a limited "budget"—specifically a small response window—MDLMs often suffer from budget-induced premature commitment.

The authors found that when space is tight, models tend to fix the final answer (the "tail" of the response) near the window boundary before they have finished the intermediate reasoning steps. Once these answer tokens are fixed, they cannot be changed and act as context for all subsequent steps, often leading the model down a path of flawed reasoning to justify a premature conclusion.

Figure 1: A smaller window (blue) leads to lower accuracy and longer, "rambling" responses compared to a larger window (orange).

Methods

The researchers introduce HiLo-RD, a self-distillation approach that doesn't require an external teacher model. Instead, it uses the model's own "wide-thinking" self to guide its "lean-reasoning" self.

  1. Paired States: A frozen "teacher" model processes a prompt with a large generation window (e.g., 1024 tokens). A trainable "student" model (the same model with LoRA adapters) processes the same prompt but with a "cropped" small window (e.g., 512 tokens).
  2. Representation Matching: The student is trained to minimize the Mean Squared Error (MSE) between its hidden representations and the teacher's representations at positions the teacher has already committed.
  3. Efficient Trajectory Acquisition: To speed up training and inference, they implement a "clean EOT" stopping rule, which terminates decoding as soon as a complete, valid response is extracted, avoiding unnecessary computation.
Figure 5: Overview of the HiLo-RD distillation process.

Datasets

The authors evaluated HiLo-RD across several challenging benchmarks:

  • GSM8K: Grade school math word problems (primary training and evaluation set).
  • MATH500: A subset of the more difficult MATH dataset.
  • HumanEval & MBPP: Python programming benchmarks used to test out-of-domain generalization (zero-shot transfer from math training).

Key Insights and Ideas

  • The "Cramped" Attention Trap: Attention diagnostics revealed that in low-budget settings, models focus heavily on the response-window boundary before fixing the first tail tokens. Expanding the window relaxes this focus, allowing for better reasoning. Figure 2: Boundary-focused attention in low-budget settings.

  • The Distraction of Early Answers: Once an answer is prematurely committed, the model's attention shifts toward it and away from the reasoning prefix. This feedback loop forces the model to "hallucinate" reasoning that fits the pre-selected (and often wrong) answer. Figure 3: Attention shifts toward the committed tail, distracting from reasoning.

  • Postponing the "Aha!" Moment: A simple intervention called "Tail Delay"—manually preventing the model from committing tail tokens early—recovered correct answers in over 43% of previously failed cases. This proved that the model could find the right path if it just waited.

  • Self-Evolving Reasoning: The authors showed that HiLo-RD can be applied iteratively. Each cycle of distillation further improves the quality-budget trade-off, creating a "self-evolving" model that gets better and faster using the same data.
Figure 2: Boundary-focused attention in low-budget settings.
Figure 3: Attention shifts toward the committed tail, distracting from reasoning.

Key Results

HiLo-RD achieved impressive gains across the board, realizing the goal of "Think Wide, Reason Lean":

  • GSM8K: Accuracy rose from 68.84% to 78.24% (+9.4 percentage points) while the average response length dropped by ~25% (from 362.8 to 271.3 tokens).
  • MATH500: Accuracy improved from 28.40% to 32.40%.
  • Cross-Domain Generalization: Despite being trained only on math, the model improved on the MBPP coding benchmark, reaching a Pass@1 of 41.40% (up from 38.60%) and producing significantly more concise code (84.7 vs 151.0 tokens).
  • Efficiency: The combination of distillation and fast-rollout stopping rules provided a much better quality-latency trade-off than vanilla models or other post-training methods like GRPO.
Figure 6: HiLo-RD (squares) achieves higher accuracy at lower latency than vanilla models (circles).

Limitations and Future Work

While highly effective, the authors note some areas for future exploration:

  • Decoding Specificity: The "clean EOT" rule and commitment logic are tailored to the MDLM architecture; adapting these to other non-autoregressive frameworks might require tweaks.
  • Intermediate Supervision: The current method focuses on final hidden states. Incorporating richer intermediate supervision throughout the denoising trajectory could potentially sustain improvements over even more distillation cycles.
  • Task Dependency: The degree of response compression achieved was task-dependent (e.g., much stronger on MBPP than HumanEval), suggesting that some domains are naturally more "compressible" than others.

Summary updated

Abstract

We propose HiLo-RD (High-to-Low Budget Representation Distillation) to improve masked diffusion language models (MDLMs) under a limited generation budget. Unlike autoregressive models, MDLMs choose both what to generate and where to commit it. This creates two coupled challenges: predicting useful content before reasoning is complete, and deciding when to fix answer tokens that shape later predictions. We identify budget-induced premature commitment: even when a correct solution fits, the model can fix answer fragments near the window boundary before resolving intermediate steps. These fragments become context for later predictions and redirect subsequent reasoning. Existing post-training methods do not directly transfer the stronger trajectories that the same model follows with a larger window. HiLo-RD addresses this gap by (1) pairing large-window teacher states with cropped small-window views and (2) matching their hidden representations at committed positions. This supervision guides both token prediction and commitment order without an external teacher. Across mathematics benchmarks, HiLo-RD improves low-budget accuracy and shortens responses; these gains extend to code generation without code-domain training. On GSM8K, accuracy rises by 9.4 percentage points as mean readable length falls by about 25%. Attention diagnostics and fixed-budget order interventions support the proposed mechanism. Together, these results show that large-window reasoning can improve small-window decoding.
BibTeX
@misc{li2026hilo,
  author = {Xiang Li and Chengting Yu and Jungang Li and Yilun Zheng and Junzhuo Li and Xingyao Zhang and Wenbo Su and Xuming Hu},
  title = {{HiLo-RD: Cross-Budget Self-Distillation for Masked Diffusion Language Models}},
  year = {2026},
  eprint = {2610.36901},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.36901},
}
Figure