Language-model agents normally start every task from scratch. Their weights are frozen, so a mistake made yesterday is repeated today. A popular fix is experiential memory: after each task the agent writes down what worked, as a reusable routine, and retrieves relevant routines on later tasks. Agent Workflow Memory (AWM) raised WebArena success from 23.5% to 35.5% this way. Yet the same literature shows that language models are easily thrown off by irrelevant text in their prompt. A retrieved memory is text the model did not ask for, written about a different page and a different moment. This proposal starts from a simple suspicion. Past experience helps an agent decide what to do next, but it can hurt the agent's judgement of what is actually in front of it: which buttons exist, whether the task is even possible, and what the exact arguments are. The reason is that stale specifics from old episodes act as distractors.
We propose to measure this effect separately at three levels of the task and then fix it with a specific mechanism. Each memory entry is split into two parts. The first is an abstract plan skeleton, which is always offered. The second is a set of concrete "grounding anchors", meaning exact element descriptions, argument formats, and observed preconditions. An anchor is shown to the agent only after a cheap check confirms that it matches the current observation. The two-sentence version is this: memory should be gated by level, with abstractions supplied freely for planning and specifics supplied only when the live environment confirms them. We expect this to keep the planning gains of memory while removing its perception and execution harms, especially when the website changes between visits.
Core Claim
The claim has two parts, and either can fail on its own. The first part is diagnostic. Break agent failures into a three-level task hierarchy. Level 1 covers perception and solvability, meaning whether the referenced element or tool exists and whether the task is feasible. Level 2 covers subgoal planning, meaning loops, stalls, and wrong ordering. Level 3 covers grounded execution, meaning malformed actions and wrong argument values. Under this breakdown, flat experiential memory should improve Level 2 while increasing Level 1 and Level 3 errors whenever retrieved memories come from near-miss tasks or pages whose layout has changed.
The second part is methodological. Gating concrete memory content on a match with the current observation should recover the Level 1 and Level 3 losses without giving back the Level 2 gains. It should also beat gating triggered by errors.
This framing also offers a way to reconcile a contradiction in the file. Diagnosing Retrieval vs. Utilization Bottlenecks in LLM Agent Memory finds that retrieval precision almost fully predicts accuracy in conversational QA (r=0.98), with utilization failures rare. Recuris finds that in long-horizon tool tasks, read recall stays stable while write execution collapses. Our hypothesis is that in procedural tasks the retrieved memory can be correct at the plan level while its concrete residue misleads action generation. Retrieval quality therefore does not reveal whether the memory helped. That is a utilization failure specific to acting, and QA benchmarks cannot expose it.
Relation to Prior Work
Three systems motivate and constrain the design.
Agent Memory Distillation (AMD) is the closest method. It distills a teacher's trajectories into workflow, subtask, and function tiers. Workflow and subtask memories are injected up front, and function-level hints are injected only after a tool error. It reports +27.2 points on AppWorld. We differ in three ways:
- Our memory is induced online from the agent's own runs, whereas AMD's is frozen and teacher-made.
- Our trigger for releasing specifics is a positive match against the current observation, whereas AMD's trigger is a failure that has already happened.
- We treat the solvability judgement as a tier that memory can corrupt, which AMD does not address.
Recuris couples a verified to-do list with a skill library and localizes failures to components. Its verification relies on programmatic environment checkers. Our gate needs only the observation the agent already receives. Recuris also does not tier its skill library.
HiAgent chunks and summarizes the agent's working context within a single episode, doubling success on long AgentBoard tasks (42% vs 21%). It keeps nothing across episodes. We treat it as complementary and plan to stack it with our method.
Memora supplies the representational idea we reuse: high-level abstractions paired with fine-grained cue anchors. Memora uses anchors as retrieval hooks for dialogue memory. We repurpose them as execution-time content whose release is conditioned on the live environment.
The noise literature supplies the mechanism we expect.
- Less is More shows that incidental nouns in a problem statement steer code models toward the wrong data structure. Our hypothesis is that stale element names in a memory do the same. - Improving Zero-shot Reader shows reader accuracy falling as irrelevant documents accumulate.
Two papers constrain the design. Diagnosing Bottlenecks shows that summarization can be lossy enough to underperform raw chunks. For that reason we do not throw away specifics; we withhold them until they are confirmed. ToolBeHonest and WebArena show that solvability judgement is fragile in both directions. Models hallucinate tools when context mentions a familiar environment, and GPT-4 wrongly declared 54.9% of feasible WebArena tasks impossible. This is why solvability is a first-class tier in our measurements.
The tier structure itself follows hierarchical evaluation practice. AgentBoard contributes subgoal-level Progress Rate, and ToolBeHonest separates solvability, planning, and missing-tool analysis.
Research Program
Step 1: tier-attribution tool. For each failed episode we assign the first irrecoverable error to one of the three levels, using rules over the trace:
- Level 1: an action references an element or tool absent from the current observation, or the agent makes a feasibility claim that contradicts ground truth.
- Level 2: repeated subgoals, stalls, or skipped subgoals, measured against AgentBoard's human subgoal annotations where they exist.
- Level 3: an action that is well-targeted but malformed or carries a wrong argument.
An LLM judge handles traces the rules cannot classify. We validate the whole labeller against a few hundred hand-labelled traces, reporting agreement before any conclusion is drawn.
Step 2: memory store. Memory is induced online, AWM-style, from episodes an automatic evaluator judges successful. Each entry stores two things:
- A parameterized plan skeleton, with concrete values replaced by variables as in AWM.
- A list of grounding anchors, each consisting of an element descriptor (role, label text, nearby structure), an argument format, and the precondition under which it was used.
Failed episodes also produce entries. Those entries carry only anchors describing what was absent, so that later feasibility decisions can consult them.
Step 3: gate. At each step the skeleton of the retrieved memory stays in the prompt. A candidate anchor is released only if its descriptor matches an element in the current observation above a similarity threshold. The matcher is a small cross-encoder in the spirit of Mind2Web's element filter. Unmatched anchors are withheld and replaced by a one-line notice that the page differs from past experience. That notice is itself a signal to explore rather than replay.
Past-failure anchors may inform an infeasibility judgement only if their precondition matches too. This rule targets the WebArena-style false-infeasibility error, where the agent decides a task is impossible because an older episode failed.
Baselines
The naive reference is the same agent, prompt, and model with no memory. This anchors every tier-level delta. On WebArena, AWM reports a 23.5% success rate for its no-memory baseline.
The strongest existing method on the headline benchmark is online AWM, which injects flat workflows proactively. Its reported numbers are 35.5% success on WebArena and 8.9 to 14.0 absolute points over baselines in cross-domain Mind2Web (Agent Workflow Memory). We re-run it rather than copy numbers. The re-run uses the same backbone models (one open-weight model and one API model), the same task order for online induction, the same evaluator, and the same retrieval budget. Beating it means higher end-to-end success on unperturbed WebArena, or at least parity there. It also means a statistically significant gain on the perturbed split, together with lower Level 1 and Level 3 attributed failure rates.
The most dangerous competitor is our own construction, which we call AWM+AMD schedule. It uses the same online-induced two-part memory, but concrete anchors are released only after an execution error, as in AMD. AMD reports +27.2 points on AppWorld with its offline version. This baseline isolates our key ingredient: observation-matched release versus error-triggered release. If it matches us, the novelty collapses to online induction alone. We report that result either way.
A retrieval-strength baseline stores raw successful trajectories and retrieves them with hybrid search plus reranking. Diagnosing Bottlenecks found this the best configuration on LoCoMo, at 81.1% accuracy. It tests whether better retrieval alone removes the problem.
A summary baseline stores only LLM-written summaries of past episodes, testing the "just compress it" view.
On AgentBoard we add HiAgent alone and HiAgent combined with our memory, to show whether the cross-episode gate adds to intra-episode chunking. HiAgent's reported 42% vs 21% average success on its five long-horizon tasks is the reference.
Two further ablations complete the set: one keeps skeletons but removes anchors entirely, and one releases all anchors unconditionally. Every condition gets the same token and call budget, with the matcher's cost counted against ours.
Why not just…?
Why not just improve retrieval, since retrieval precision predicted accuracy almost perfectly? That result comes from conversational QA, where a retrieved fact is either the answer or not. In our setting a memory can be the right task and still carry wrong specifics, because the page has changed. Better retrieval would fetch the right workflow more reliably and still inject stale element names. The hybrid-rerank baseline tests this directly. If it closes the Level 1 and Level 3 gap on the perturbed split, our hypothesis is wrong and retrieval really is the whole story.
Why not just abstract everything, as AWM already does by replacing values with variables? Variable substitution removes task values like product names, but workflows still name interface elements and action formats. Stripping those too is the lossy-compression trap documented in Diagnosing Bottlenecks: the agent would lose exactly the details that make execution cheap. The skeleton-only ablation measures this. We expect it to reduce Level 1 harm but raise Level 3 errors and step counts relative to gated anchors.
Why not just release details reactively after an error, as AMD does? This is the real threat. Error-triggered release costs nothing extra and already works offline. Its weakness in principle is that some memory-induced errors never raise an error signal: a false "task impossible" verdict, or a click on a plausible but wrong element. We therefore compare the two release rules specifically on silent-failure categories, as well as overall. If the AMD schedule matches us there too, we will report that our contribution is the online, tier-diagnosed version of AMD rather than a new gating rule.
Why not just use a long context and include all past trajectories? HiAgent's flat-context baseline reached 21% against 42% with chunking, and AgentBoard documents the "6-step wall". We include only a capped version of this setting, because the evidence against it is strong.
Proposed Experiments
Clean benchmarks. We run WebArena, cross-domain Mind2Web, and the long-horizon AgentBoard tasks. We report success rate, AgentBoard Progress Rate, steps, tokens, and tier-attributed failure rates.
Perturbed WebArena split. We construct a perturbed split of WebArena by modifying replayed sites in three ways: inserting modal pop-ups, renaming button labels, and reordering navigation. Memory is induced on the clean sites and tested on the perturbed ones. No reported result was found on Lacuna for any memory method under such perturbations, so all numbers here are new. The split directly tests the known rigidity of AWM.
Distractor stress test. We replace the top retrieved memory with a near-miss memory from a sibling task on a different site. We then measure how each method's success and tier profile degrade as the fraction of near-miss memories rises.
Solvability check. We evaluate on ToolBeHonest and on WebArena's unachievable tasks, measuring false-feasibility and false-infeasibility rates separately. The ToolBeHonest references are GPT-4o at 37.0 and Gemini-1.5-Pro at 45.3 on its 0–100 score. Our question there is not the absolute score but whether memory of past tool use makes solvability hallucination worse, and whether gating prevents it.
Decisive checks. Three results decide the project.
- The diagnostic claim is supported only if flat AWM shows a measurable rise in Level 1 or Level 3 failures relative to no memory on the perturbed or near-miss conditions. If that harm does not appear, gating has nothing to fix, and we report the negative result.
- The method claim is supported only if observation-gated release beats the AWM+AMD error-triggered schedule on perturbed success and on silent Level 1 errors, at matched cost.
- The reconciliation claim is supported if tier attribution shows the retrieved memory was plan-correct (Level 2 subgoals matched) in most episodes where execution failed. That pattern would explain why Recuris sees stable reads alongside collapsing writes.
References
- Agent Workflow Memory
- Diagnosing Retrieval vs. Utilization Bottlenecks in LLM Agent Memory
- Recursive Experiential-Working Memory Evolution for Long-Horizon Agent Harnesses
- Agent Memory Distillation: Empowering Small LLM Agents with Hierarchical Teacher Memory
- HiAgent: Hierarchical Working Memory Management for Solving Long-Horizon Agent Tasks with Large Language Model
- Memora: A Harmonic Memory Representation Balancing Abstraction and Specificity
- Less is More: Summary of Long Instructions is Better for Program Synthesis
- Improving Zero-shot Reader by Reducing Distractions from Irrelevant Documents in Open-Domain Question Answering
- ToolBeHonest: A Multi-level Hallucination Diagnostic Benchmark for Tool-Augmented Large Language Models
- WebArena: A Realistic Web Environment for Building Autonomous Agents
- AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents
- Mind2Web: Towards a Generalist Agent for the Web