Database-Diff Step Filtering for Agent Distillation

When a small language-model agent is trained by copying a larger model's tool-use transcripts, much of what it copies is wrong. Some steps break things, some do nothing, and some narrate a success that never happened ("I've sent the payment") while the payment app's database is unchanged. Current recipes handle this in one of two coarse ways. They keep only whole transcripts that pass a final programmatic check, or they ask another language model to judge each step, and that judge itself makes mistakes. This proposal takes a third route for environments where the software state is a real database. We snapshot the database after every action and compare it with the snapshot before. Each step is then labeled by what it actually changed, and the training loss is switched off on steps whose effect was harmful, and on any narration that contradicts the recorded change.

Put plainly, the environment's own records become a step-by-step referee for imitation learning. That referee is objective, cheap, and blind to how plausible the agent's prose sounds. The idea also raises a second question that prior work leaves unsettled: should the student still see the bad steps in its training context? A transcript full of failed calls and false claims is exactly the kind of distracting context known to degrade language models. We test whether replacing those steps with short, verified "nothing changed" notes reduces the student's tendency to build on its own false premises.

Core Claim

In database-backed agent environments, step labels derived from per-step state differences ("diffs") are more accurate than a language-model critic. They are also the only label source that can catch state-claim hallucinations, meaning narration that asserts a change the database contradicts. We claim that masking the imitation loss with these labels yields a stronger distilled agent than whole-transcript filtering or critic-based step filtering. We expect the gains to show up most clearly in two places: fewer false claims of completion, and fewer unintended side effects ("collateral damage"). We further claim that how failed steps are presented in context matters separately from whether the student is trained on them.

The closest work frames the gap. Step Rejection Fine-Tuning (SRFT) keeps failed software-engineering transcripts in context but zeroes the loss on steps that a Claude 4 Sonnet critic labels harmful. It raises SWE-bench Verified resolution from 30.9% (whole-transcript rejection fine-tuning) to 32.2%. Its critic reaches only 0.737 F1 on good steps, and it never inspects environment state. Three systems do use programmatic state checks, but only to accept or reject whole rollouts: VeriEnv checks backend state through a software development kit (SDK), Echoverse uses SQL diffs, and Agent-Diff defines before/after "State-Diff Contracts" but only for evaluation. DiDPO is the nearest in spirit: it splits code patches into sub-diffs to assign credit. However, it operates on git text diffs inside online reinforcement learning, not on database records inside offline imitation. Our contribution sits in the empty cell. We use step-level, state-grounded labels for loss masking, and we add an explicit check of the agent's claims against those labels.

How Steps Are Labeled

For each training rollout we record a database snapshot $S_t$ after every action $a_t$ and compute the record-level diff $\Delta_t = S_t \ominus S_{t-1}$. For successful rollouts, and for tasks where the benchmark's unit tests define the required end state, we also have a target diff $\Delta^\ast$. This is the set of record changes the task requires.

A write step is labeled productive if its changes fall inside $\Delta^\ast$ and persist to the end. It is collateral if it touches records outside $\Delta^\ast$ that the benchmark forbids changing. It is reverted if a later step undoes it. It is failed if it produced an error or an empty diff while the agent intended a write.

Read-only steps change nothing, so the diff cannot judge them directly. We label them by data flow. A read counts as useful if a value it returned appears in the arguments of a later productive write. Otherwise it is neutral and left unmasked. This is the weakest part of the labeler, and we measure it separately.

Separately, a lightweight extractor parses the agent's reasoning text at each step for assertions about state ("created", "sent", "deleted", "task complete"). Each assertion is matched against $\Delta_t$ and the cumulative diff. Any span that asserts a change the diff does not contain is a state-claim hallucination. This failure was documented qualitatively in OS-Copilot, where agents report completion over untouched systems.

Training then uses a masked imitation loss:

$$\mathcal{L} = -\sum_{t} m_t \log \pi_\theta(y_t \mid h_t)$$

Here $y_t$ is the step's output (reasoning plus action), $h_t$ is the history shown to the student, and $m_t \in \{0,1\}$ is set to zero on collateral, reverted, and failed steps. Inside otherwise-kept steps, the per-token mask is zeroed on the hallucinated claim spans.

Research Program

We build the labeler on AppWorld. Its nine apps run over 101 database tables with about 370,000 rows. Its tasks average 42.5 API calls and are graded by state-based unit tests that also check for collateral damage, so per-step diffs are cheap and the target state is well defined.

A strong teacher model generates multiple rollouts per training task. We keep every rollout, successful or not, label every step, and distill into student models at roughly 4B and 8B scale. The pipeline needs one extra database snapshot per step and no additional model calls. This contrasts with SRFT's frontier-model critic and with the roughly 420,000 simulator calls reported by LLM4Cov.

The context question is studied as a separate factor. SRFT shows harmful steps to the student so that it learns to tolerate noisy history. Two lines of evidence point the other way. LLM4Cov found that a memoryless view (current draft plus latest error) beats full history, and Agent-Diff found that exhaustive documentation dilutes gains on complex tasks (+2.4, not significant, versus +7.0 for relevant documentation). ACON compresses agent histories, but only at inference time. We compare three ways of presenting masked steps in training context: verbatim, deleted, or replaced by a one-line verified stub generated from the diff (for example, "attempted payment; database unchanged; error: insufficient balance"). Masking decides what the student learns to produce. The context treatment decides which premises it learns to condition on. Our hypothesis is that verbatim false claims in context teach the student to treat its own unverified narration as fact, which is the error-snowballing pattern seen in multi-step generation.

Baselines

The naive reference is naive distillation on every rollout with no filtering. Above it sits whole-transcript rejection fine-tuning on unit-test-passing rollouts only, which is the recipe used by VeriEnv and Echoverse. VeriEnv reports +6.06 points for Qwen3-4B and +9.70 for LLaMA-3.2-3B-Instruct on WebArena with this recipe. Both baselines use the same teacher rollouts, student initialization, token budget, and hyperparameter search budget as our method.

The strongest offline competitor is SRFT itself, reimplemented on AppWorld with a frontier language-model critic labeling each step as Good, Harmful, or Unnecessary, and with the same loss masking and verbatim context. It is the right opponent because it answers the same question: which steps of imperfect transcripts to learn from. Its reported figure is 32.2% resolved on SWE-bench Verified, against 30.9% for rejection fine-tuning. We found no reported result on Lacuna for SRFT or any step-filtered imitation method on AppWorld. Beating SRFT means higher Task Goal Completion on both AppWorld test splits (Test-Normal and Test-Challenge), over three seeds, by more than seed variance, together with lower collateral-damage and false-completion rates.

As an upper reference, not a matched competitor, we report the GVPO result from Group Verification-based Policy Optimization. GVPO is online reinforcement learning with penalties for intermediate execution failures. A 32B model trained with it reaches 49.4% Task Goal Completion on AppWorld Test-Challenge, versus 36.7% for o1, and GPT-4o with ReAct prompting solves 48.8% of Normal and 30.2% of Challenge tasks. We do not expect a small offline-distilled student to match the 32B online result. We do report what fraction of that gap our method closes at a stated fraction of GVPO's environment interactions.

Two ablations isolate the key ingredient. The first is exception-only masking, which zeroes loss on steps that raised an error and nothing else; it captures the cheap part of GVPO's signal. The second is diff labels without claim-span masking. A third arm, critic plus diff (masking a step if either source flags it), tests whether the two signals are complementary.

Why not just…?

Why not just keep only successful transcripts? It is simpler and the label is exact. But SRFT's analysis shows that about 76% of steps in failed transcripts are valid, so discarding them throws away most of the hard-task data. Successful transcripts can also contain detours and claim hallucinations that a pass/fail check cannot see. The comparison against whole-transcript rejection fine-tuning, together with a breakdown of gains on tasks the teacher rarely solves, settles this directly.

Why not just use an LLM critic, as SRFT does? This is the real threat. A strong critic sees intent and can judge read steps, which diffs cannot. We address it on two fronts. First, we hand-label about 300 AppWorld steps and compare diff labels, critic labels, and their union against that ground truth, reporting per-category F1 alongside SRFT's 0.737 figure. Second, the head-to-head training comparison shows whether any label advantage carries over to the trained agent. If the critic matches the diff labeler on downstream success, the remaining case for our method is cost and the claim-hallucination check, and we will say so.

Why not just mask steps that throw exceptions? GVPO's finding that execution failures are informative suggests this cheap heuristic may capture most of the signal. It cannot see writes that succeed but touch the wrong records, writes that are later reverted, or false narration over a successful call. These are precisely the collateral and hallucination categories. The exception-only ablation tests this. If it comes within one point of full diff labeling on Task Goal Completion and on collateral damage, the richer labeler is not justified, and we will report that outcome.

Why not just compress context at inference time, as ACON does? Compression at inference can remove distracting history, but it does not change which behaviors the weights learned. We run ACON-style compression on top of each trained student, so we can see whether training-time stubs and inference-time compression are redundant or additive.

Proposed Experiments

All arms are trained on the same AppWorld training-split rollouts and evaluated on Test-Normal and Test-Challenge for Task Goal Completion. Alongside it we report two state-grounded metrics: the collateral-damage rate from AppWorld's side-effect tests, and the false-completion rate, meaning the fraction of episodes where the agent declares success while the unit tests fail. Zero-shot transfer is measured on Agent-Diff, whose 224 enterprise API tasks have their own state-diff contracts. Nine models score between 38.0 and 88.1 there, so this is a test of whether the student learned grounded behavior rather than AppWorld-specific habits. As a safety check, we run AgentDojo, where GPT-4o's targeted attack success rate is 47.7% under the "Important message" attack. The concern is that state-grounded training should not make the agent more willing to carry out injected writes.

The study has four decisive checks. The first is label accuracy against human annotation, split by write versus read steps. The second is the main comparison against SRFT-style critic masking at matched budget. The third is the exception-only ablation, which determines whether the richer diff labels are worth building. The fourth is the context-treatment factorial (verbatim, deleted, stub). Its key prediction is that stubs lower false-completion rate without lowering Task Goal Completion; if verbatim context instead wins, that supports SRFT's view that exposure to failures teaches recovery. A result in which diff labels beat the critic on false completions and collateral damage, but not on overall success, would still be informative. It would mean state grounding mainly buys reliability rather than capability, and we would report it that way.