Machine Unlearning for Masked Diffusion Language Models

Georu Lee, Seungwon Jeong, Hoki Kim, Jinseong Park, Woojin LeearXiv 2026 · arXiv

TL;DR

As Masked Diffusion Language Models (MDLMs) emerge as powerful alternatives to traditional autoregressive LLMs, they bring a unique challenge: how do we make them forget? This paper introduces Masked Diffusion Unlearning (MDU), the first unlearning framework tailored for MDLMs. By steering conditional denoising trajectories back toward a prompt-masked unconditional anchor, MDU effectively erases targeted knowledge while preserving the model's structural fluency and general utility.

What Problem Does This Paper Address?

With the rapid deployment of large language models, the ability to selectively remove sensitive, copyrighted, or incorrect data post-training—known as machine unlearning—has become a regulatory and ethical necessity.

While numerous unlearning techniques exist for standard autoregressive (AR) models, they are fundamentally mismatched with Masked Diffusion Language Models (MDLMs) like LLaDA and Dream. Unlike AR models that generate text sequentially from left to right, MDLMs generate text by iteratively denoising masked positions in parallel, utilizing bidirectional context. Directly applying AR unlearning objectives to MDLMs ignores this parallel denoising structure, often leading to catastrophic model collapse or incomplete forgetting. This paper fills this gap by introducing an unlearning framework built natively for the diffusion paradigm.

Methods

The authors propose Masked Diffusion Unlearning (MDU), which conceptualizes unlearning as reversing the trajectory shift that occurs during fine-tuning. During fine-tuning, an MDLM learns specific facts by shifting its denoising trajectory away from a prompt-masked unconditional distribution (a reference state where the prompt is hidden) toward a prompt-conditional distribution.

MDU reverses this by minimizing the forward Kullback-Leibler (KL) divergence from the prompt-conditional prediction to a temperature-scaled, prompt-masked unconditional anchor at every masked response position:

$$L_{\text{forget}}(\theta) = \mathbb{E}_{(x,y)\sim \mathcal{D}_f, t, y_t} \left[ \frac{1}{|M_t|} \sum_{i \in M_t} \text{KL}\left( p_\theta^c(\cdot \mid x, y_t) \parallel \frac{1}{Z_i} p_{\theta_0}^u(\cdot \mid m, y_t)^\tau \right) \right]$$

A temperature hyperparameter ($\tau \in [0, 1]$) allows fine-grained control over the privacy-utility trade-off:

  • $\tau = 1$: Targets the exact unconditional anchor, removing prompt-conditioned facts while preserving natural syntax and language priors.
  • $\tau = 0$: Forces a uniform distribution (Maximum Entropy), which provides aggressive forgetting but risks over-forgetting.
Figure 1: Overview of the proposed Masked Diffusion Unlearning (MDU).

Datasets

MDU was evaluated on two open-source MDLM backbones (LLaDA-8B-Instruct and Dream-7B-Instruct) across two standard benchmarks:

  • TOFU: A synthetic dataset of fictitious author profiles used to measure exact forgetting and retention across four splits (forget, retain, real authors, and world facts).
  • RWKU: A benchmark targeting real-world knowledge removal for public figures, evaluating forget memorization, neighbor retention, and downstream utility (MMLU, TruthfulQA, TriviaQA).

Key Insights and Ideas

  • Unlearning as Trajectory Collapse: Instead of introducing an artificial forgetting distribution, unlearning in diffusion models can be framed as collapsing a specific conditional trajectory back onto a marginal reference (null) trajectory.
  • Token-Role Sensitivity: By categorizing response tokens into in-context (reused from the prompt), structural (syntax/grammar), and stored-knowledge (the actual facts), the authors discovered that the conditional-anchor KL divergence naturally spikes at stored-knowledge tokens.
  • Selective Suppression: MDU leverages this property to primarily reshape memorized associations. It reduces the KL divergence of stored-knowledge tokens by ~20% while leaving structural tokens virtually untouched (+1.4%), allowing the model to forget facts without losing its ability to generate fluent text.
Figure 3: Token-level conditional-anchor KL analysis along a representative TOFU forget-query trajectory under the Base SFT model, showing KL values along the denoising trajectory.

Key Results

  • Superior Forget-Retain Trade-off: On the TOFU benchmark, MDU achieved near-zero answer probabilities on the forget set while maintaining significantly higher retain-set performance than AR-adapted baselines like Gradient Ascent (GA) or Negative Preference Optimization (NPO).
  • Prevention of Textual Collapse: Standard unlearning baselines adapted to MDLMs frequently cause the model to degenerate into repetitive token loops (e.g., repeating words like "persever" or "that"). MDU avoids this entirely, yielding either clean refusals ("Unfortunately, I do not have any specific information...") or fluent, plausible alternative text.
Qualitative results showing MDU avoiding repetitive loops.

Limitations and Future Work

  • Training vs. Inference Gap: MDU operates on randomly sampled masked denoising states during training. However, actual inference in MDLMs utilizes a structured, iterative denoising trajectory with specific sampling and remasking strategies. Bridging this gap between training-time objectives and inference-time generation is a key next step.
  • Evaluation Scale: Future work will need to evaluate MDU on larger forget sets, scale it to even larger MDLM backbones, and test its resilience against advanced adversarial privacy attacks.

Summary updated

Abstract

Recent masked diffusion language models (MDLMs), such as LLaDA and Dream, have achieved performance comparable to autoregressive large language models. Unlike autoregressive models, which generate text sequentially, MDLMs generate text by iteratively denoising masked positions in parallel. During fine-tuning, MDLMs learn to recover responses from masked response states conditioned on a prompt, thereby shifting their predictions from a prompt-masked unconditional distribution toward a prompt-conditional distribution. Despite this distinct generative and fine-tuning mechanism, machine unlearning for MDLMs remains largely unexplored. In this paper, we propose Masked Diffusion Unlearning (MDU), the first unlearning framework for MDLMs, by revisiting the process of learning specific knowledge in terms of diffusion. Specifically, MDU minimizes a forward KL divergence from the prompt-conditional prediction to a prompt-masked unconditional anchor at every masked response position, with a temperature scaling parameter to control the privacy-utility trade-off. Our empirical results on standard benchmarks and MDLM backbones show that MDU achieves high unlearning performance compared to existing LLM unlearning methods. Code is available at https://github.com/leegeoru/MDU.
BibTeX
@misc{lee2026machine,
  author = {Georu Lee and Seungwon Jeong and Hoki Kim and Jinseong Park and Woojin Lee},
  title = {{Machine Unlearning for Masked Diffusion Language Models}},
  year = {2026},
  eprint = {2605.18253},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2605.18253},
}
Figure