TL;DR
TimeROME-DLM is the first training-free, inference-time knowledge editing and unlearning framework designed specifically for Masked Diffusion Language Models (MDLMs). By shifting the causal tracing paradigm from token positions to the temporal denoising trajectory, it achieves highly effective fact erasure and modification with zero weight updates, zero extra VRAM, and up to a 14x wall-clock speedup over gradient-based alternatives.
What Problem Does This Paper Address?
As Masked Diffusion Language Models (MDLMs) like LLaDA and Dream emerge as powerful alternatives to traditional autoregressive (AR) large language models, they bring a unique challenge: how do we edit their knowledge or make them "unlearn" specific facts?
In AR models, popular locate-then-edit methods (like ROME or MEMIT) trace facts to specific token positions and early MLP layers. However, MDLMs do not generate text left-to-right; instead, they iteratively de-mask tokens across the entire sequence in a data-dependent order. Consequently, traditional positional hot-spots disappear. Furthermore, trying to apply standard gradient-based unlearning methods (like Negative Preference Optimization) to MDLMs typically results in catastrophic model collapse or no learning at all due to the extreme fragility of the diffusion mask-loss surface.
Methods
The authors introduce TimeROME-DLM, a completely training-free framework that operates entirely at inference time via two key components:
- Temporal Indirect Effect (TIE) Causal Tracing: Instead of tracking information across token positions, TIE tracks it across the denoising trajectory (diffusion steps). By running three parallel diffusion passes—a clean pass, a corrupted pass (adding Gaussian noise to subject token embeddings), and a patch pass (restoring clean activations at specific coordinates)—TIE isolates exactly when and where a fact becomes recoverable.
- Multi-Fact Low-Rank Edit Memory: Once the optimal coordinate is located, the target facts are compiled offline into a closed-form, ridge-regularized least-squares update. This update is injected via a PyTorch forward hook at the identified layer during every denoising step. To prevent this edit from corrupting unrelated knowledge, a Top-$q$ sparsification filter ensures the edit only fires on tokens closely aligned with the target subject.
Datasets
The framework was rigorously evaluated across several established unlearning and knowledge-editing benchmarks:
- TOFU (forget01, forget05, forget10): Fictitious author biographies used to measure exact unlearning and utility retention.
- RWKU: A real-world knowledge unlearning benchmark.
- MUSE (Books and News splits): Used to evaluate verbatim and knowledge memorization transfer.
- WHP (Harry Potter completions): Testing the erasure of highly memorized fiction.
- WMDP-Bio: Evaluating the removal of hazardous biological knowledge.
- lm-evaluation-harness: A suite of 12 standard downstream tasks to monitor general model utility.
Key Insights and Ideas
- Knowledge is Temporal, Not Spatial, in MDLMs: Causal tracing reveals that factual associations in MDLMs localize heavily within a specific band of lower-to-mid layers ($\ell \approx 8\text{–}19$) during early-to-middle denoising steps ($k \approx 1\text{–}2$), rather than being tied to a fixed token position.
- The Residual Stream is the Messenger: Unlike AR models where MLP layers act as primary key-value memories, the dominant causal effects in MDLMs are carried directly within the residual stream.
- Inference-Time Hooks Bypass Fine-Tuning Fragility: By solving a closed-form linear system and applying it dynamically during generation, TimeROME-DLM completely avoids the massive activation memory overhead (tens of gigabytes of VRAM) and optimization instability associated with backpropagation through iterative diffusion steps.
Key Results
- Superior Forget Strength: On the TOFU forget01 dataset using LLaDA-8B, TimeROME-DLM reduced the forget-set log-probability by 83.24 nats, outperforming the strongest gradient baseline (AlphaEdit) by 7x and MEMIT by over 20x.
- Massive Efficiency Gains: The entire end-to-end edit construction takes just ~26 seconds for 40 facts, requiring 0 extra VRAM, compared to gradient methods that take significantly longer and demand 35–51 GB of extra activation memory.
- Cross-Architecture Transferability: A single hyperparameter configuration successfully transferred across six distinct MDLM backbones (including LLaDA-8B-Instruct, Dream-7B, and the sparse MoE LLaDA-MoE-1.4B) without requiring re-tuning.
- Excellent Utility Preservation: Sequential editing tests showed that the model could absorb 50 facts one-by-one while keeping the retain-set log-probability nearly flat (within $\sim 1$ nat).
Limitations and Future Work
- Neighbor Selectivity: On the RWKU dataset, the simple penalty term struggled to perfectly isolate the target fact from its close semantic neighbors, causing slight collateral unlearning.
- In-Context Relearning Vulnerability: Like many forget-only methods, the edit's effectiveness degrades (by $\sim 8$ nats) if the forgotten answer is explicitly provided in-context immediately before the query.
- Global vs. Per-Fact Coordinates: The current implementation selects a single global (layer, module) coordinate for the entire forget set; moving to per-fact coordinate selection could yield even cleaner edits.
Summary updated