Replay-Localized Attribution of Agent Failures to Fine-Tuning Data

When a tool-using language agent fails a customer, today's tools can say that it failed, and sometimes at which turn. They cannot say which training examples taught it to fail that way. This proposal asks that question in a setting where it can be answered and checked.

We fine-tune an open model into a customer-service agent on a training corpus we control. We then expose it to realistic, uncooperative simulated users, meaning language-model "customers" that are impatient, vague, or reluctant to share details. Finally, we trace the failures those users expose back to specific fine-tuning examples. The tracing uses influence estimation: a family of methods that approximate how much a model's behavior on one input would change if a particular training example were removed.

The core bet is that the user simulator is more than an evaluation harness. It is the instrument that makes attribution well-posed. Because a simulated conversation can be rewound to any turn and replayed with a different agent action, we can find the exact decision that caused the failure and the alternative action that would have rescued it. We then attribute that margin between the failing and the rescuing action, rather than the whole failed transcript.

The opportunity is concrete. Agents that look strong against polite simulators collapse against realistic ones. Beyond Cooperative Simulators evolves such users and shows that training against them buys robustness. τ-bench reports GPT-4o reaching only 35.2% pass^1 in its airline domain, with consistency over eight repeated trials falling below 25%. Meanwhile, scalable attribution now runs at 7B scale and beyond. Tracing LLM Social Reasoning to Pretraining Data applies gradient attribution to OLMo3-7B and confirms its findings by unlearning the influential data. No one has pointed these tools at interactive failures, and doing so naively is likely to fail for a specific reason that this proposal is built around.

Core Claim

Gradient attribution applied directly to a failed multi-turn transcript will mostly retrieve training examples that share the transcript's format: other tool-calling dialogues in the same domain. It will not retrieve the examples that taught the bad decision. Two ingredients should fix this.

The first ingredient is replay localization. The simulator lets us branch the conversation at each agent turn. At each turn we resample agent actions from a stronger reference agent and keep the earliest turn $k$ from which some alternative action leads to task success under the same simulated user. This gives an interventional estimate of the critical turn. It replaces the language-model judge used by TRAJDEBUG, which reaches only 34.11% macro accuracy at critical-error detection.

The second ingredient is a contrastive measurement function. We define the quantity to attribute as

$$f = \log \pi_\theta(a^{-} \mid h_k) - \log \pi_\theta(a^{+} \mid h_k),$$

where $\pi_\theta$ is the agent's policy with parameters $\theta$, $h_k$ is the conversation history up to the critical turn, $a^{-}$ is the action the agent actually took, and $a^{+}$ is the replay-verified rescuing action. Attributing $f$ cancels gradient directions shared by any well-formed action in that context, such as JSON formatting, domain vocabulary, and politeness. What remains is the direction that separates, say, "call the cancellation tool now" from "ask which reservation the user means."

The claim is falsifiable. Removing the training examples ranked highest by this score and retraining should restore robustness against uncooperative users more than removing examples ranked by text similarity, by plain transcript attribution, or at random. It should do so without lowering performance against cooperative users.

Why This Is Different From What Exists

The closest work splits into three groups, and each covers one side of the gap.

The first group traces capabilities to data on static benchmarks. Tracing LLM Social Reasoning to Pretraining Data aggregates attribution scores over 576 pretraining-corpus bins for SocialIQA and MMLU. It supplies our attribution machinery and its unlearning-style validation. However, it never touches an interactive agent, and it explicitly disclaims claims about simulation fidelity. We move from macro-level capability profiles to individual decision-level failures. We also move from pretraining to the fine-tuning stage, where ground-truth counterfactual retraining is affordable.

The second group attributes within interaction but stays local. A Snapshot of Influence adapts TracIn (a method that sums training-step gradient similarities) to the rollout buffer of a single reinforcement-learning round. It does this precisely because the agent's own actions shape its training data. That circularity motivates our choice to attribute the fine-tuning stage, where the dataset is static. The simulated conversations are only queries, so the circular dependency does not arise. DUET uses influence functions inside a data-mixture search driven by scalar feedback on static tasks. It shows that attribution can steer data choice. It never models where in an interaction things went wrong.

The third group localizes failures in transcripts but never reaches training data. TRAJDEBUG finds that agents repair about 62% of their local mistakes. This finding constrains our design: the first error is usually the wrong attribution target, which is why we localize by replay rather than taking the first deviation.

The theoretical caution in Barriers to Counterfactual Credit Attribution for Autoregressive Models concerns compounding probability shifts across generated sequences. Our measurement function conditions on a fixed history and scores a single action span. This deliberately sidesteps the compounding problem rather than solving it. We will report how much of the failure our single-turn margin explains, measured by whether retraining shifts the final outcome or only the local action.

Research Program

We first build a controlled agent-training corpus. It mixes three sources. The first is agent trajectories generated by a strong agent against the default cooperative τ²-bench simulator, on retail and airline tasks disjoint from the evaluation tasks. The second is general single-turn instruction data, such as Alpaca, which RapidIn and In-Context Probing also use. The third is a small planted subset of trajectories that deliberately demonstrate premature tool calls before required arguments are elicited. This is the "action bias" documented by Can Tool-augmented LLMs be Aware of Incomplete Conditions?. The planted subset gives a known ground-truth cause, in the spirit of RapidIn's backdoor-retrieval checks, where it reports auROC near 1.0.

We fine-tune OLMo3-7B with low-rank adapters, and a smaller open model with full-parameter training. We choose OLMo3 so that a later extension can attribute to its open pretraining corpus with the tooling from the second seed paper.

We then generate failures. Each evaluation task is run under the default simulator and under Persona Policies users from Beyond Cooperative Simulators. We keep tasks that the agent passes cooperatively but fails against personas, so the failure is persona-induced rather than a missing capability. For each failure we run replay localization to obtain $h_k$, $a^{-}$, and $a^{+}$.

For attribution, we compute per-example training gradients once and compress them with the factorized sparse projection of GraSS/FactGraSS. This choice is a scaling constraint, not a contribution. Following First is Not Really Better Than Last, we restrict gradients to middle-layer attention value projections and combine layers by positional voting. We also run a full-layer variant, because that paper's finding has only been tested on classification. The query gradient is $\nabla_\theta f$ for each localized failure. Scores are aggregated by failure category, using persona type and failure kind (premature action, rule violation, unelicited assumption), to produce a ranked list of training examples per category.

Finally, we validate by counterfactual retraining. We remove or downweight the top-ranked examples at several budgets, retrain with multiple seeds, and re-evaluate. Multiple seeds are needed because Distributional Training Data Attribution shows that single-example effects can drown in seed noise. We also inspect what the retrieved examples have in common. One concrete hypothesis comes from LLMs Get Lost In Multi-Turn Conversation, which reports a 39% average drop when instructions arrive in pieces, along with premature first-turn commitment. The hypothesis is that single-turn instruction examples rewarding immediate complete answers are disproportionately responsible for persona-induced failures.

Baselines

Naive reference points. The do-nothing reference is the fine-tuned agent itself, evaluated against both simulators. The naive data intervention removes the same number of training examples uniformly at random. Every retraining comparison uses identical removal budgets, the same base model and hyperparameters, the same seeds, and the same evaluation tasks and persona pool. Beating the random baseline means a persona-condition pass^1 gain whose seed-level confidence interval excludes the random-removal gain.

Text-similarity retrieval. BM25 and dense-embedding retrieval use the localized failing turn as the query and remove the most similar examples. This is the cheapest serious competitor. TRAK separates fact tracing from behavior tracing. UnTrac shows that gradient-similarity heuristics are fooled by surface format. Both results are evidence that similarity retrieval will miss behavior-teaching data, but neither tested agents. It gets the same removal budgets.

Plain transcript attribution. This is the strongest existing method applied as practitioners would use it: TrackStar-style gradient attribution, as in the second seed paper, scored on the loss of the full failed transcript, with the same compressed gradient store. Beating it isolates the value of our two ingredients.

Ablations isolating the key ingredients. Two ablations separate the ingredients. The first keeps the contrastive margin but localizes with TRAJDEBUG's judge or the last agent turn instead of replay. The second keeps replay localization but drops the contrast and attributes only $\log \pi_\theta(a^{-} \mid h_k)$.

On recovery of the planted cause, the method should rank planted examples above both baselines. On the hard metric, persona-condition task success after removal, it should exceed plain transcript attribution with cooperative-condition success unchanged within noise.

Reference results. There is no reported result on Lacuna for any attribution method on τ-bench or τ²-bench, so the relevant reference numbers are task success rates. These are GPT-4o's τ-bench pass^1 of 61.2% (retail) and 35.2% (airline), and the second seed paper's report of a 17% relative task-success improvement from training on persona-conditioned interactions.

Why not just…?

Why not just retrieve similar training dialogues with BM25 or embeddings? It might work when the failure mirrors a training example's wording, as with the planted subset. That is exactly why the planted subset is included: it tells us whether similarity succeeds on easy cases. The interesting failures are behavioral, such as committing before clarifying, and TRAK's behavior-tracing results suggest lexical neighbors will be format matches. If similarity retrieval matches our method on retraining gains, we will report that attribution adds no practical value for this failure class.

Why not just train on persona-simulated conversations and skip diagnosis? This is the real threat. The second seed paper already shows that augmentation improves robustness. We run augmentation at a matched number of added versus removed examples, and also combined with our removal. If augmentation alone matches removal plus augmentation, the contribution narrows to explanation: knowing which data sources cause fragility, which matters for curating future corpora and for models one cannot cheaply re-fine-tune. We will state plainly whether removal adds anything beyond that diagnostic role.

Why not just localize with TRAJDEBUG and attribute that turn? TRAJDEBUG's 34.11% accuracy and 42.1% trigger-miss rate suggest that judge-based localization will often attribute the wrong decision. The first ablation tests this directly. Replay is also more expensive, so if the judge-localized variant performs as well, we will recommend it.

Decisive Checks

Three outcomes would decide the project.

The first is planted-cause recovery. Precision at k for the planted premature-tool-call trajectories, under each attribution method, tells us whether the pipeline can find a cause we know exists.

The second is counterfactual retraining on τ²-bench retail and airline. We remove 0.5%, 2%, and 5% of the corpus by each ranking, with three or more seeds, and report pass^1 and pass^k under persona users and cooperative users. The project succeeds if replay-contrastive attribution yields the largest persona-condition gain at a fixed budget without a cooperative-condition loss.

The third is a transfer check on the sharded multi-turn suite from LLMs Get Lost In Multi-Turn Conversation. If removing data selected from τ²-bench persona failures also shrinks the sharded-versus-full-instruction gap, the attributed data encode a general premature-commitment habit rather than a benchmark quirk.

A negative result is informative here too. If contrastive scores match random removal on retraining, it suggests that static, linearized attribution cannot capture the compounding dynamics of conversations, at least at this scale. That is the tension the research file flags between interactive failures and attribution methods that assume a fixed final model.

Workshop authors to connect with