DebugLM: Learning Traceable Training Data Provenance for LLMs

Wenjie Jacky Mo, Qin Liu, Xiaofei Wen, Wenxuan Zhou, Zhe Zhao, Muhao Chen

Abstract

Large language models (LLMs) are trained through multi-stage pipelines over heterogeneous data sources, yet developers lack a principled way to pinpoint the specific data responsible for an observed behavior. This lack of observability reduces debugging to reactive patching and makes failures prone to recur under distribution shift or subsequent model updates. To address this limitation, we propose DebugLM, a framework that equips LLMs with built-in data provenance, enabling them to explicitly trace the origins of their behaviors to specific training data sources. Specifically, the model learns to associate its responses with unique provenance tags that indicate the responsible dataset, empowering developers to precisely identify where undesirable behaviors are learned. Building on this capability, DebugLM further supports targeted test-time remediation, enabling developers to selectively trigger targeted refusal for specified data sources without retraining or modifying model parameters. Experiments demonstrate that DebugLM provides accurate behavior tracing in multi-stage training pipelines and effective test-time remediation while preserving the general utility of the model.

1. Introduction

Modern Large Language Models (LLMs) are the product of complex, multi-stage training regimes—including pre-training, supervised fine-tuning (SFT), and reinforcement learning from human feedback (RLHF)—applied to massive, heterogeneous datasets. While this process yields high-performance models, it creates a significant observability gap. When a model exhibits undesirable behavior, such as generating harmful instructions or hallucinating medical advice, developers often lack a principled method to identify the specific training data responsible.

Current post-hoc attribution methods, such as influence functions, are computationally prohibitive at the scale of billions of parameters and often fail when data mixtures shift across training stages. This lack of transparency forces developers into a cycle of "reactive patching," fixing symptoms without diagnosing the underlying cause. DebugLM addresses this by internalizing data provenance directly within the model's parameters, providing a "paper trail" for every response.

2. The DebugLM Framework

DebugLM treats data provenance as a first-class citizen during the training process. The framework is designed around three core pillars: internalizing provenance tags, a dual-mode training objective, and test-time remediation.

Overview of the DebugLM framework, illustrating the multi-stage training pipeline, data tracing, and test-time remediation.

2.1 Source-Tagging and Debug Interface

Every dataset in the training pipeline is assigned a unique, single-token provenance tag (e.g., <TOFU>, <WMDP>). To access this information, DebugLM introduces a lightweight debug interface triggered by a special <DEBUG> token. In standard operation, the model behaves normally. However, when the debug trigger is prepended to a prompt, the model enters "debug mode" and appends the relevant provenance tag to its output, explicitly identifying the source dataset that influenced the generation.

2.2 Dual-Mode Training

The model is trained using a joint objective that balances standard next-token prediction with provenance association. This ensures that the model learns to associate specific response patterns with their corresponding source tags without degrading its general utility. Unlike lexical matching methods, DebugLM's parametric tracing allows it to identify sources even when the output format is significantly altered (e.g., converting a free-form answer into a multiple-choice response).

2.3 Test-Time Remediation

A key advantage of DebugLM is its ability to perform "surgical" safety interventions. By prepending a "quarantine tag" to a prompt, developers can force the model to refuse queries that draw from specific, problematic datasets. This remediation occurs at inference time without requiring expensive retraining or weight modifications, effectively isolating hazardous knowledge while maintaining performance on benign tasks.

3. Experimental Evaluation

The framework was evaluated using a diverse set of benchmarks, including TOFU (fictitious author profiles), ChatDoctor (medical interactions), TruthfulQA (misconceptions), Beavertails (safety/toxicity), and WMDP (hazardous knowledge).

3.1 Tracing Accuracy and Robustness

DebugLM achieved a Tracing Success Rate (TSR) exceeding 95% across various datasets. This performance is comparable to running a dedicated 1B-parameter external classifier, but without the additional operational overhead. Furthermore, the system demonstrated remarkable robustness; in semantic perturbation tests where questions were rewritten by GPT-4o, DebugLM maintained a 100% success rate in identifying the correct data source.

3.2 Utility and Scaling

Importantly, the addition of diagnostic capabilities does not impose a "utility tax." Performance on standard benchmarks like MMLU and ARC-Challenge remained stable. The system also proved scalable, successfully managing up to 199 unique author tags in the TOFU dataset, suggesting it can handle high-resolution provenance beyond simple dataset-level labels.

4. Discussion and Conclusion

DebugLM provides a proactive solution for LLM observability, enabling precise diagnosis and remediation of model failures. While highly effective, the approach is currently limited to models where the developer has control over the training phase, as it cannot be applied post-hoc to frozen black-box models. Future research will explore scaling provenance to millions of document-level tags and investigating whether self-reporting mechanisms can be used to automatically filter hallucinated data during training. By making training data traceable, DebugLM moves LLM development toward a more transparent and manageable engineering discipline.

Summary updated

Abstract

Large language models (LLMs) are trained through multi-stage pipelines over heterogeneous data sources, yet developers lack a principled way to pinpoint the specific data responsible for an observed behavior. This lack of observability reduces debugging to reactive patching and makes failures prone to recur under distribution shift or subsequent model updates. To address this limitation, we propose DebugLM, a framework that equips LLMs with built-in data provenance, enabling them to explicitly trace the origins of their behaviors to specific training data sources. Specifically, the model learns to associate its responses with unique provenance tags that indicate the responsible dataset, empowering developers to precisely identify where undesirable behaviors are learned. Building on this capability, DebugLM further supports targeted test-time remediation, enabling developers to selectively trigger targeted refusal for specified data sources without retraining or modifying model parameters. Experiments demonstrate that DebugLM provides accurate behavior tracing in multi-stage training pipelines and effective test-time remediation while preserving the general utility of the model.
BibTeX
@misc{mo2026debuglm,
  author = {Wenjie Jacky Mo and Qin Liu and Xiaofei Wen and Wenxuan Zhou and Zhe Zhao and Muhao Chen},
  title = {{DebugLM: Learning Traceable Training Data Provenance for LLMs}},
  year = {2026},
  eprint = {2603.17884},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2603.17884},
}

Loading related content...

Figure