TL;DR
As Masked Diffusion Language Models (MDLMs) emerge as powerful alternatives to traditional autoregressive LLMs, they bring a unique challenge: how do we make them forget? This paper introduces Masked Diffusion Unlearning (MDU), the first unlearning framework tailored for MDLMs. By steering conditional denoising trajectories back toward a prompt-masked unconditional anchor, MDU effectively erases targeted knowledge while preserving the model's structural fluency and general utility.
What Problem Does This Paper Address?
With the rapid deployment of large language models, the ability to selectively remove sensitive, copyrighted, or incorrect data post-training—known as machine unlearning—has become a regulatory and ethical necessity.
While numerous unlearning techniques exist for standard autoregressive (AR) models, they are fundamentally mismatched with Masked Diffusion Language Models (MDLMs) like LLaDA and Dream. Unlike AR models that generate text sequentially from left to right, MDLMs generate text by iteratively denoising masked positions in parallel, utilizing bidirectional context. Directly applying AR unlearning objectives to MDLMs ignores this parallel denoising structure, often leading to catastrophic model collapse or incomplete forgetting. This paper fills this gap by introducing an unlearning framework built natively for the diffusion paradigm.
Methods
The authors propose Masked Diffusion Unlearning (MDU), which conceptualizes unlearning as reversing the trajectory shift that occurs during fine-tuning. During fine-tuning, an MDLM learns specific facts by shifting its denoising trajectory away from a prompt-masked unconditional distribution (a reference state where the prompt is hidden) toward a prompt-conditional distribution.
MDU reverses this by minimizing the forward Kullback-Leibler (KL) divergence from the prompt-conditional prediction to a temperature-scaled, prompt-masked unconditional anchor at every masked response position:
$$L_{\text{forget}}(\theta) = \mathbb{E}_{(x,y)\sim \mathcal{D}_f, t, y_t} \left[ \frac{1}{|M_t|} \sum_{i \in M_t} \text{KL}\left( p_\theta^c(\cdot \mid x, y_t) \parallel \frac{1}{Z_i} p_{\theta_0}^u(\cdot \mid m, y_t)^\tau \right) \right]$$
A temperature hyperparameter ($\tau \in [0, 1]$) allows fine-grained control over the privacy-utility trade-off:
- $\tau = 1$: Targets the exact unconditional anchor, removing prompt-conditioned facts while preserving natural syntax and language priors.
- $\tau = 0$: Forces a uniform distribution (Maximum Entropy), which provides aggressive forgetting but risks over-forgetting.
Datasets
MDU was evaluated on two open-source MDLM backbones (LLaDA-8B-Instruct and Dream-7B-Instruct) across two standard benchmarks:
- TOFU: A synthetic dataset of fictitious author profiles used to measure exact forgetting and retention across four splits (forget, retain, real authors, and world facts).
- RWKU: A benchmark targeting real-world knowledge removal for public figures, evaluating forget memorization, neighbor retention, and downstream utility (MMLU, TruthfulQA, TriviaQA).
Key Insights and Ideas
- Unlearning as Trajectory Collapse: Instead of introducing an artificial forgetting distribution, unlearning in diffusion models can be framed as collapsing a specific conditional trajectory back onto a marginal reference (null) trajectory.
- Token-Role Sensitivity: By categorizing response tokens into in-context (reused from the prompt), structural (syntax/grammar), and stored-knowledge (the actual facts), the authors discovered that the conditional-anchor KL divergence naturally spikes at stored-knowledge tokens.
- Selective Suppression: MDU leverages this property to primarily reshape memorized associations. It reduces the KL divergence of stored-knowledge tokens by ~20% while leaving structural tokens virtually untouched (+1.4%), allowing the model to forget facts without losing its ability to generate fluent text.
Key Results
- Superior Forget-Retain Trade-off: On the TOFU benchmark, MDU achieved near-zero answer probabilities on the forget set while maintaining significantly higher retain-set performance than AR-adapted baselines like Gradient Ascent (GA) or Negative Preference Optimization (NPO).
- Prevention of Textual Collapse: Standard unlearning baselines adapted to MDLMs frequently cause the model to degenerate into repetitive token loops (e.g., repeating words like "persever" or "that"). MDU avoids this entirely, yielding either clean refusals ("Unfortunately, I do not have any specific information...") or fluent, plausible alternative text.
Limitations and Future Work
- Training vs. Inference Gap: MDU operates on randomly sampled masked denoising states during training. However, actual inference in MDLMs utilizes a structured, iterative denoising trajectory with specific sampling and remasking strategies. Bridging this gap between training-time objectives and inference-time generation is a key next step.
- Evaluation Scale: Future work will need to evaluate MDU on larger forget sets, scale it to even larger MDLM backbones, and test its resilience against advanced adversarial privacy attacks.
Summary updated