When a language-model agent is fine-tuned on one tool domain after another, it forgets earlier domains. The usual fix is experience replay: keep a small buffer of old examples and mix them into new training batches. In agent work, two things are usually measured and stored at the level of whole episodes, scored only by whether the task finished. We think forgetting in agents is often not of that kind. An agent can keep "finishing" old tasks while it quietly stops doing what made those finishes correct. It may skip a prerequisite state change, call a forbidden tool, or take a long detour. Outcome-level forgetting scores will not show this, and replaying whole successful episodes chosen at random may not repair it.
The proposal is to replace that by a loop. Every so often, the agent re-runs a small set of old-domain probe tasks. A trajectory evaluator scores each probe step by step, checking which required subgoals (milestones) are still reached and which forbidden actions (minefields) are now triggered. The agent then rehearses short, stored trajectory segments that cover exactly the milestones and minefields that regressed. In two sentences: use process-level trajectory evaluation to diagnose which sub-skills and constraints a continually trained agent has forgotten, then replay only the milestone-bounded segments that fix them. We test whether this beats equal-budget random replay, and whether outcome-only forgetting metrics underestimate constraint forgetting.
Core Claim
We make two claims, and each can fail on its own. The measurement claim is that, under sequential fine-tuning across tool domains, the drop in the rate of avoiding forbidden actions and reaching milestones is larger and earlier than the drop in final task success. If so, outcome-based backward transfer, the standard forgetting metric, systematically understates forgetting in agents. The method claim is that replay chosen by measured per-milestone regression and stored as milestone-bounded segments recovers process-level performance better than random replay, outcome-filtered replay, or loss-prioritized replay at a matched training-token budget.
If random replay closes the process gap as well as targeted replay, we will report that as the main finding. That result would still be useful: it would mean trajectory evaluation matters for measuring agent forgetting but not for choosing what to rehearse.
How This Relates to Prior Work
The trajectory-evaluation side supplies the scoring machinery.
- ToolSandbox scores tool-use trajectories against human-authored graphs of milestones and minefields across 11 domains. It ships the per-step labels we need, but it never trains or studies forgetting.
- Mobile-Bench does the analogous thing with logical CheckPoints over action histories.
- Towards Policy-Compliant Agents shows that compliance must be judged over action sequences and prefixes. It trains a fast detector, PolicyGuard-4B, which reports 90.1% accuracy at 22.5 ms per example.
We use all three only as scorers. None of them asks whether a trained agent keeps passing these checks over time.
BEACON constrains and motivates the segmentation design. It shows that once a milestone is reached, later progress depends on the current state, not on how the agent got there. That property is what lets us store a segment between two milestones without its prefix. BEACON uses it for credit assignment in single-domain reinforcement learning, not for continual retention.
The replay side supplies the training recipe and the strongest skeptical prior.
- Continual Learning with Tiny Episodic Memories shows that replaying even one stored example per class beats regularization methods by 7–17%. This is why random replay is a serious competitor and not a strawman.
- Dark Experience Replay improves replay by matching the old model's output scores (logits) instead of only the labels.
- SuRe prioritizes replay of sequences on which the model is most surprised, measured by high negative log-likelihood.
Our selection signal differs from SuRe's. It is behavioral and grounded in the environment, rather than token loss on stored text, which may not track whether the agent still calls the right API in the right order.
DISTR is the closest replay design. It prioritizes replay by a task's "vulnerability" to policy noise. However, it works on continuous robot control, ranks whole tasks, and keeps only trajectories that succeeded at the end. We rank individual milestones and minefields by measured regression on symbolic tool trajectories. t-DGR argues that state coverage matters more than temporal order in replay. Stateful APIs, where an out-of-order call fails outright, make that doubtful for agents, so we keep each segment's internal order intact.
The closest agent-side works do not train weights continually.
- AgentHER relabels failed trajectories offline as demonstrations of whatever they did achieve. It keeps whole traces, detours and violations included, and has no forgetting dynamics.
- PATH-Bench shows that piling up experience in an in-context memory makes 5 of 8 agents drift below their starting performance. Its gatekeeper chooses what to retrieve into the prompt, never what to train on.
- PolicyBank evolves an external text memory of policy rules.
We study forgetting in the weights, which none of these addresses.
Research Program
The setting is sequential fine-tuning of an open 7B-class agent over a fixed order of tool domains. Hermes-2-Pro-Mistral-7B is a natural choice, since ToolSandbox reports it trailing GPT-4o by over 40 points.
Training data. For each domain, we collect rollouts from a stronger teacher model and from the agent itself on that domain's training scenarios. We keep those that pass the trajectory evaluator, and fine-tune with supervised learning before moving to the next domain.
The buffer. The buffer holds milestone-bounded segments, not episodes. Each segment is the stretch of a compliant trajectory between two consecutive milestones, stored with the environment state at its start. Each is tagged with the milestones it achieves and the minefields it passes near without triggering.
For every minefield we also store a contrastive pair: a prefix that led to a violation in some rollout, paired with the compliant next action. These pairs serve as boundary examples, in the spirit of Retrospective Adversarial Replay, but drawn from real trajectories instead of synthesized.
The probe step. After every fixed number of training steps, the agent runs a small held-out probe set from each earlier domain. The evaluator then estimates, for each milestone or minefield $m$, the current pass rate $\hat p_m$, where passing means reaching the milestone or avoiding the minefield. Segments are sampled for replay with probability proportional to
$$w_m = \max(0,\; \hat p_m^{\,\text{ref}} - \hat p_m^{\,\text{now}}) + \epsilon,$$
Here $\hat p_m^{\,\text{ref}}$ is the pass rate right after that domain was learned, $\hat p_m^{\,\text{now}}$ is the latest probe estimate, and $\epsilon$ is a small floor so that items not yet regressed are still occasionally rehearsed.
Replayed segments get the ordinary next-token loss. Following Dark Experience Replay, they may also get a logit-matching term against scores stored when the segment was learned.
Noise control. Probe estimates are noisy. ToolSandbox's simulated user alone has roughly an 8% error rate. We therefore smooth $\hat p_m$ over probe rounds and report how stable the priority ranking is across seeds.
Second testbed. The same loop runs on the ALFWorld → WebShop → ScienceWorld sequence. There, milestones come from the environment-feedback detectors BEACON already defines, so the method is not tied to one benchmark's hand-authored graphs.
Where segments do not apply. On the web domains of PolicyGuardBench, minefield detection is done by PolicyGuard-4B rather than hand-written rules. This tests whether a learned checker is accurate enough to drive replay. Those web domains lack milestone graphs, so there the method reduces to minefield-driven contrastive replay.
Baselines
Naive reference points. The do-nothing reference is plain sequential fine-tuning with no replay. Its outcome and process scores after the final domain set the floor. Multi-task joint training on all domains at once sets the ceiling, the performance continual training is trying to approach.
Strongest competitor: equal-budget random replay. This is reservoir-sampled replay of whole successful episodes, as in Continual Learning with Tiny Episodic Memories. Its prior record is the strongest of any baseline, and nothing in the file shows it failing on agent trajectories.
Fairness is the crux, because our method spends compute on probe rollouts that random replay does not. Random replay therefore gets the same number of replayed training tokens and the same buffer size in tokens. It also gets the probe-rollout compute converted into extra replay steps. All methods share one learning-rate and replay-ratio sweep of the same size, tuned on a separate domain order.
Other competitors. Outcome-filtered episode replay keeps only finished trajectories, as DISTR does, but without its diffusion generator. SuRe-style replay prioritized by negative log-likelihood is run with its slow/fast averaged-weights variant. It is the strongest published replay prioritizer for language models, with 78.0% average accuracy reported on its LNT benchmark in SuRe. DER++-style logit replay is applied to whole episodes.
Reference numbers. No reported result found on Lacuna for any replay method on sequential ToolSandbox or ALFWorld/WebShop/ScienceWorld domain sequences. The only anchors are single-domain ones: GPT-4o's 73.0 similarity score on ToolSandbox, and BEACON's 92.9% ALFWorld, 75.6% WebShop and 45.3% ScienceWorld success rates from BEACON. We use these only as upper context, not as targets.
What counts as beating random replay. The bar is a statistically reliable improvement across five seeds and two domain orders on two measures: final-domain-averaged minefield-avoidance rate and milestone pass rate. Outcome success must also not fall below random replay's.
Ablations. Three ablations isolate the ingredients. Diagnosis-weighted selection over whole episodes removes segmentation. Segment replay with uniform weights removes diagnosis. Removing the contrastive minefield pairs tests whether forbidden-action knowledge needs negative evidence at all.
Why not just…?
Why not just replay random successful episodes? This is the real threat. Continual Learning with Tiny Episodic Memories shows that tiny random buffers are hard to beat. A successful ToolSandbox episode already contains the prerequisite state changes and avoids minefields, so random replay may preserve process behavior for free.
The equal-budget comparison and the "uniform-weight segments" ablation settle it. If random episodes match targeted segments on minefield avoidance, we drop the selection mechanism and keep only the measurement result. Our expectation that it will not match rests on one point: minefields are rarely approached in successful episodes, so random replay gives little gradient near the violation boundary. The contrastive-pair ablation tests that point directly.
Why not just prioritize by model surprise, as SuRe does? Loss on stored text measures how unlikely the stored tokens are under the current model. It does not tell whether the agent, acting in a live environment, still reaches the milestone. A trajectory can stay low-loss while the agent's own rollouts wander off it after one early error. Comparing SuRe-prioritized replay with diagnosis-prioritized replay at equal budget tests this. We will also report the rank correlation between per-segment surprise and per-milestone regression. If that correlation is high, surprise is a cheaper proxy, and we will say so.
Why not just keep a memory in the prompt instead of training? PATH-Bench finds that ungated in-context experience causes 5 of 8 agents to decline. PolicyBank reports that ordinary memory baselines fall to near zero on its policy-gap tasks. Retrieval is still a legitimate alternative, though, so we include a retrieval agent that puts the same buffer contents into the context of the sequentially trained model. If it matches weight replay on process metrics, the value lies in the stored segments, not in training on them.
Why not just add a KL penalty to the original model? A KL-to-reference penalty constrains how far the model's output distribution moves from a reference model, here the base model. That base model never knew the later domains. Continual GUI Agents reports its plain reinforcement-fine-tuning baseline, which carries such a penalty, at only 17.7% on the ScreenSpot-Pro resolution sequence. We include a KL-to-previous-checkpoint baseline, the stronger version of this idea, so that it is not dismissed by construction.
Proposed Experiments and Decisive Checks
Setup. The primary benchmark is ToolSandbox, with its domains split into training scenarios and held-out probe and test scenarios per domain. Domain orders are fixed in advance. The second is ALFWorld/WebShop/ScienceWorld with BEACON's milestone detectors. The third, minefield-only, is the PolicyGuardBench web domains with PolicyGuard-4B as the checker.
Metrics. Every method is scored at both the outcome and the process level. Outcome metrics are final task success and outcome backward transfer. Process metrics are milestone pass rate, minefield violation rate and process backward transfer, the same transfer statistic computed over milestone and minefield pass rates. We also report replayed tokens and wall-clock time, including probe rollouts.
Decisive checks. Three checks decide the outcome.
- Measurement. Under sequential fine-tuning with no replay, process backward transfer must be significantly more negative than outcome backward transfer, and it must turn negative at an earlier domain. If it does not, the premise fails and the method is unnecessary.
- Method. At matched compute, diagnosis-weighted segment replay must beat random episode replay on minefield avoidance on both testbeds. The ablations must attribute the gain to diagnosis, segmentation or contrastive pairs, rather than to a tuning artifact.
- Learned checker. On the PolicyGuardBench domains, replay driven by PolicyGuard-4B must keep most of the gain from replay driven by exact rules. This answers whether manual milestone authoring, the bottleneck ToolSandbox itself flags, is needed for the approach to scale.
References
- ToolSandbox: A Stateful, Conversational, Interactive Evaluation Benchmark for LLM Tool Use Capabilities
- Mobile-Bench: An Evaluation Benchmark for LLM-based Mobile Agents
- Towards Policy-Compliant Agents: Learning Efficient Guardrails for Policy Violation Detection
- Milestone-Guided Policy Learning for Long-Horizon Language Agents
- Continual Learning with Tiny Episodic Memories
- Dark Experience for General Continual Learning: a Strong, Simple Baseline
- SuRe: Surprise-Driven Prioritised Replay for Continual LLM Learning
- Stable Continual Reinforcement Learning via Diffusion-based Trajectory Replay
- t-DGR: A Trajectory-Based Deep Generative Replay Method for Continual Learning in Decision Making
- AgentHER: Hindsight Experience Replay for LLM Agent Trajectory Relabeling
- PATH-Bench: Path-Dependent Evaluation of Lifelong Agents
- PolicyBank: Evolving Policy Understanding For Evolving Agents
- Retrospective Adversarial Replay for Continual Learning
- Continual GUI Agents