TimeROME-DLM: Temporal Causal Tracing and Low-Rank Inference-Time Knowledge Editing for Masked Diffusion Language Models

Zhengtao Yao, Liuyang Song, Hongbo Zhang, Chenhao Wei, Haoyan Xu, Guang Yang, Siheng WangarXiv 2026 · arXiv

TL;DR

TimeROME-DLM is the first training-free, inference-time knowledge editing and unlearning framework designed specifically for Masked Diffusion Language Models (MDLMs). By shifting the causal tracing paradigm from token positions to the temporal denoising trajectory, it achieves highly effective fact erasure and modification with zero weight updates, zero extra VRAM, and up to a 14x wall-clock speedup over gradient-based alternatives.

What Problem Does This Paper Address?

As Masked Diffusion Language Models (MDLMs) like LLaDA and Dream emerge as powerful alternatives to traditional autoregressive (AR) large language models, they bring a unique challenge: how do we edit their knowledge or make them "unlearn" specific facts?

In AR models, popular locate-then-edit methods (like ROME or MEMIT) trace facts to specific token positions and early MLP layers. However, MDLMs do not generate text left-to-right; instead, they iteratively de-mask tokens across the entire sequence in a data-dependent order. Consequently, traditional positional hot-spots disappear. Furthermore, trying to apply standard gradient-based unlearning methods (like Negative Preference Optimization) to MDLMs typically results in catastrophic model collapse or no learning at all due to the extreme fragility of the diffusion mask-loss surface.

Methods

The authors introduce TimeROME-DLM, a completely training-free framework that operates entirely at inference time via two key components:

  1. Temporal Indirect Effect (TIE) Causal Tracing: Instead of tracking information across token positions, TIE tracks it across the denoising trajectory (diffusion steps). By running three parallel diffusion passes—a clean pass, a corrupted pass (adding Gaussian noise to subject token embeddings), and a patch pass (restoring clean activations at specific coordinates)—TIE isolates exactly when and where a fact becomes recoverable.
  2. Multi-Fact Low-Rank Edit Memory: Once the optimal coordinate is located, the target facts are compiled offline into a closed-form, ridge-regularized least-squares update. This update is injected via a PyTorch forward hook at the identified layer during every denoising step. To prevent this edit from corrupting unrelated knowledge, a Top-$q$ sparsification filter ensures the edit only fires on tokens closely aligned with the target subject.
Overview of the TimeROME-DLM pipeline, showing how facts are mapped to the denoising trajectory and injected via a low-rank edit memory hook.

Datasets

The framework was rigorously evaluated across several established unlearning and knowledge-editing benchmarks:

  • TOFU (forget01, forget05, forget10): Fictitious author biographies used to measure exact unlearning and utility retention.
  • RWKU: A real-world knowledge unlearning benchmark.
  • MUSE (Books and News splits): Used to evaluate verbatim and knowledge memorization transfer.
  • WHP (Harry Potter completions): Testing the erasure of highly memorized fiction.
  • WMDP-Bio: Evaluating the removal of hazardous biological knowledge.
  • lm-evaluation-harness: A suite of 12 standard downstream tasks to monitor general model utility.

Key Insights and Ideas

  • Knowledge is Temporal, Not Spatial, in MDLMs: Causal tracing reveals that factual associations in MDLMs localize heavily within a specific band of lower-to-mid layers ($\ell \approx 8\text{–}19$) during early-to-middle denoising steps ($k \approx 1\text{–}2$), rather than being tied to a fixed token position.
  • The Residual Stream is the Messenger: Unlike AR models where MLP layers act as primary key-value memories, the dominant causal effects in MDLMs are carried directly within the residual stream.
  • Inference-Time Hooks Bypass Fine-Tuning Fragility: By solving a closed-form linear system and applying it dynamically during generation, TimeROME-DLM completely avoids the massive activation memory overhead (tens of gigabytes of VRAM) and optimization instability associated with backpropagation through iterative diffusion steps.
TIE heatmaps demonstrating that factual information concentrates in the residual stream during specific early-middle denoising steps.

Key Results

  • Superior Forget Strength: On the TOFU forget01 dataset using LLaDA-8B, TimeROME-DLM reduced the forget-set log-probability by 83.24 nats, outperforming the strongest gradient baseline (AlphaEdit) by 7x and MEMIT by over 20x.
  • Massive Efficiency Gains: The entire end-to-end edit construction takes just ~26 seconds for 40 facts, requiring 0 extra VRAM, compared to gradient methods that take significantly longer and demand 35–51 GB of extra activation memory.
  • Cross-Architecture Transferability: A single hyperparameter configuration successfully transferred across six distinct MDLM backbones (including LLaDA-8B-Instruct, Dream-7B, and the sparse MoE LLaDA-MoE-1.4B) without requiring re-tuning.
  • Excellent Utility Preservation: Sequential editing tests showed that the model could absorb 50 facts one-by-one while keeping the retain-set log-probability nearly flat (within $\sim 1$ nat).

Limitations and Future Work

  • Neighbor Selectivity: On the RWKU dataset, the simple penalty term struggled to perfectly isolate the target fact from its close semantic neighbors, causing slight collateral unlearning.
  • In-Context Relearning Vulnerability: Like many forget-only methods, the edit's effectiveness degrades (by $\sim 8$ nats) if the forgotten answer is explicitly provided in-context immediately before the query.
  • Global vs. Per-Fact Coordinates: The current implementation selects a single global (layer, module) coordinate for the entire forget set; moving to per-fact coordinate selection could yield even cleaner edits.

Summary updated

Abstract

Masked diffusion language models (MDLMs) such as LLaDA now rival autoregressive (AR) LLMs, but every existing knowledge-editing and unlearning method (ROME, MEMIT, etc.) targets AR transformers and either makes assumptions that fail under iterative denoising, or requires gradient updates whose backward-pass activations cost tens of GB of extra VRAM and which collapse MDLMs at standard learning rates. We introduce TimeROME-DLM, the first training-free, gradient-free, inference-time knowledge-editing framework for MDLMs. It couples two components: a Temporal Indirect Effect (TIE) causal-tracing protocol that identifies, for each fact, the coordinate whose intervention most strongly drives the object prediction at later denoising steps; and a closed-form, low-rank residual edit memory that aggregates subject keys and target deltas across all forget facts and applies a single ridge-regularised update at that coordinate at every diffusion forward, with sparsification to limit utility spillover. Backbone weights stay frozen; only three hyperparameters (alpha, lambda, q) are tuned on a small validation split. On TOFU forget01 with TOFU-finetuned LLaDA-8B-Base, TimeROME-DLM cuts forget-set log-probability by roughly 83 nats. The same configuration transfers to LLaDA-8B-Instruct, Dream-7B, MMaDA-8B, DiffuLLaMA-7B, and LLaDA-MoE-1.4B. It keeps retain-set log-probability nearly flat (within ~1 nat at the utility-safe operating point) across 50 sequentially inserted facts, delivers a four- to fourteen-fold wall-clock speedup with zero additional VRAM over the strongest converged training-time baseline, and scales sub-linearly to 400 facts. TimeROME-DLM closes the locate-then-edit gap between AR LLMs and MDLMs at a fraction of the computational cost.
BibTeX
@misc{yao2026timerome,
  author = {Zhengtao Yao and Liuyang Song and Hongbo Zhang and Chenhao Wei and Haoyan Xu and Guang Yang and Siheng Wang},
  title = {{TimeROME-DLM: Temporal Causal Tracing and Low-Rank Inference-Time Knowledge Editing for Masked Diffusion Language Models}},
  year = {2026},
  eprint = {2606.12841},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2606.12841},
}
Figure