Auditing Tool Traces in Agentic LLMs: Source Localization and Counterfactual Dependence Testing

Junhao Liu, Tianxiao Li, Yongxian Wu, Ziheng Duan, Ren Li, Jing Zhang, Martin Renqiang MinarXiv 2026 · arXiv

TL;DR

This paper introduces a two-stage audit to detect "shortcuts" in agentic LLMs—cases where models appear to perform complex scientific reasoning but are actually exploiting hidden patterns (like dataset identifiers) in tool outputs. By applying this to a protein-binding task, the authors show that an LLM's impressive performance was entirely dependent on leaked PDB identifiers, collapsing to chance once those IDs were redacted.

What Problem Does This Paper Address?

As we move toward "agentic" scientific workflows, we increasingly give Large Language Models (LLMs) access to external tools—simulators, databases, and analytical scripts. The outputs of these tools are serialized into a "tool trace" that the LLM uses to make a final prediction.

The danger is a specialized version of the "Clever Hans" effect: tool outputs often contain metadata, version strings, or entry identifiers (like PDB IDs) that might correlate with the ground-truth labels due to how the dataset was constructed. If an LLM learns to associate a specific ID prefix with a "positive" result, it isn't performing biological reasoning; it's just performing a lookup.

Standard benchmarks often fail to catch this because of the dilution effect: when multiple tool results are concatenated, a strong shortcut hidden in one tool can be masked by the "noise" of other tools during global screening, yet the LLM might still find and exploit it.

Methods

The authors propose a two-stage audit to unmask these shortcuts:

  1. Per-Tool TF-IDF Decomposition (PT2D): Instead of auditing the entire concatenated tool trace at once, PT2D analyzes each tool's output independently. It uses simple TF-IDF features (word and character n-grams) and a logistic regression classifier to see if any specific tool's output is "leaking" the answer. This prevents the dilution effect from hiding local shortcuts.
  2. Token-Class Redaction: Once PT2D identifies a suspect pattern (e.g., a specific ID format), the authors perform a counterfactual test. They redact that specific class of tokens from the LLM's input and re-evaluate the model. If performance drops significantly while a "control" redaction (redacting actual scientific data like peptide sequences) does not, it proves the model was relying on the shortcut.

Datasets

The study focuses on TCR–pMHC binding prediction, a critical task in immunology.

  • Source Data: A dataset from Slone et al. [2025].
  • Splits: They used "peptide-disjoint" clusters from Abeer et al. [2026] to ensure the model couldn't just memorize specific peptides. Cluster 0 was used for training/validation, and Clusters 1–5 served as the test set.
  • Tools: The LLM (Qwen3-4B) was augmented with five structural biology tools, including biophysical descriptors, retrieval from McPAS-TCR, and mutation-level stability predictions (ThermoMPNN).

Key Insights and Ideas

  • The Dilution Effect is Real: The authors demonstrated that as you add more tools to a trace, the "detectability" of a shortcut decreases for a global auditor, even though the shortcut remains just as exploitable for the LLM. Figure 2: Adding real (A) or synthetic (B) tool results dilutes Tool 3’s cross-split signal.

  • Localization is the First Step: PT2D successfully localized a massive signal to "Tool 3" (ThermoMPNN). While the full trace showed near-chance performance for a simple auditor, the isolated Tool 3 output was highly predictive. Figure 1: PT2D localizes Tool 3 as the strongest cross-split carrier.

  • Masquerading as Discovery: Without this audit, the model's AUROC of ~0.81 would look like a major success for evidence-grounded AI. The audit reveals this "success" was a mirage built on PDB ID prefixes.

Figure 2: Adding real (A) or synthetic (B) tool results dilutes Tool 3’s cross-split signal.
Figure 1: PT2D localizes Tool 3 as the strongest cross-split carrier.

Key Results

The quantitative results are striking. The fine-tuned LLM initially achieved an AUROC of 0.8136 on the test set, significantly outperforming a "no-evidence" baseline (0.4928).

However, the audit overturned this interpretation:

  • PDB-ID Redaction: When PDB identifiers were replaced with placeholders, the model's performance collapsed to 0.4972 (essentially random chance).
  • Control Redaction: When the actual peptide sequences (the intended scientific evidence) were redacted, performance remained stable at 0.8152.

This confirms the model was ignoring the biological sequences and purely exploiting the identifiers.

Table 1: Frozen-checkpoint AUROC after targeted token-class redaction

Limitations and Future Work

  • Scope: The experiments were conducted on a single (though important) benchmark and one model backbone (Qwen 4B). The authors note that dilution behavior might vary with different trace lengths and formats.
  • Post-hoc Nature: This is a diagnostic tool, not a preventative one. It tells you if your model is cheating after it has been trained.
  • Black-box Audit: The method looks at input-output behavior. It doesn't probe the internal attention mechanisms or weights of the LLM to see how it processes the shortcut.
  • Future Directions: The authors suggest that "source-level provenance analysis" should become a standard part of the pipeline before any benchmark gains in tool-augmented LLMs are trusted.

Summary updated

Abstract

Agentic LLMs increasingly support scientific prediction by grounding decisions in external evidence. In biology, they promise accurate, interpretable predictions for important problems by integrating complex experimental and computational evidence. However, serialized tool results can contain hidden label shortcuts, such as dataset identifiers or template artifacts, that create apparent success without biological reasoning. To address this risk, we introduce a two-stage audit. Stage 1, Per-Tool TF-IDF Decomposition (PT2D), localizes label-predictive patterns to individual tool results while avoiding signal dilution in the concatenated tool trace. Stage 2, Token-Class Redaction, counterfactually tests whether the frozen LLM depends on the localized shortcut. We apply this audit to TCR-pMHC binding, where tool augmentation substantially outperforms a no-evidence baseline. At face value, this suggests that the LLM has learned to use scientific evidence. The audit overturns this interpretation: PT2D localizes the strongest signal to identifiers in one tool result, and redacting them reduces performance to chance. These findings show how hidden label shortcuts can masquerade as scientific reasoning or discovery, motivating source-level audits before benchmark gains are trusted.
BibTeX
@misc{liu2026auditing,
  author = {Junhao Liu and Tianxiao Li and Yongxian Wu and Ziheng Duan and Ren Li and Jing Zhang and Martin Renqiang Min},
  title = {{Auditing Tool Traces in Agentic LLMs: Source Localization and Counterfactual Dependence Testing}},
  year = {2026},
  eprint = {2610.25540},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.25540},
}
Figure