From Reflection to Co-Evolution: Self-Evolving Agents with Locally Verified Memory

Xinxiang Yin, Xinglang Zhang, Haodong Chen, Zhiqing Cui, Weidong Tang, Yuxuan Wang, Junrui Wang, Jingyi Zhang, Jianfeng Zhang, Jian QianarXiv 2026 · arXiv

TL;DR

The paper introduces Anchor, a framework that enables LLM agents to selectively retain only the most impactful and behaviorally-aligned memories from their interactions. By using a "paired replay" test to measure how a candidate memory shifts a policy's action distribution at the source of the decision, Anchor achieves superior task performance on ALFWorld and WebShop while significantly reducing memory storage and token costs.

What Problem Does This Paper Address?

LLM agents typically improve by "reflecting" on past experiences and storing lessons in a memory bank. However, this approach faces two major hurdles:

  1. Utility Uncertainty: Just because a lesson sounds plausible doesn't mean it's useful. A memory might repeat what the agent already knows (redundancy) or, worse, reinforce a mistake.
  2. The Cost of "Memory Bloat": Storing every possible lesson leads to massive memory banks. This increases storage costs, consumes valuable context window space, and makes retrieving relevant information more expensive and difficult.

Current methods often rely on aggregate task success to judge a memory, but this fails to isolate whether a specific memory actually helped. The authors ask: Can we test a memory's value at the exact moment it was created before deciding to keep it?

Methods

The authors propose Anchor, a "Write, Test, and Reuse" framework. Its core innovation is paired replay: re-simulating a specific decision point both with and without a candidate memory to see exactly how the agent's internal probability distribution over actions changes.

The framework consists of four main components:

  • Localizer: Identifies "consequential" steps in a completed trajectory and extracts candidate memories (hints).
  • Amplituder (Magnitude Test): Uses Jensen-Shannon (JS) divergence to measure how much the memory shifts the agent's action distribution. If the shift is negligible, the memory is deemed "inert" and discarded.
  • Aspecter (Direction Test): Ensures the shift aligns with the "source role." For a successful action, the memory should reinforce it (Success Maintain). For a failed action, it should suppress the mistake or favor an alternative (Failure Correction).
  • BankMaster: Manages the memory bank, handling retrieval for future tasks and maintaining the bank by updating confidence scores or pruning stale entries.
Figure 1: Memory can change the next action. In this schematic kitchen example, putting a dirty reusable plate in the sink is preferable to the cupboard or trash bin. Anchor screens for a role-aligned change.

Datasets

The researchers evaluated Anchor using two popular LLM agent benchmarks:

  • ALFWorld: A synthetic text-based environment where agents must complete household tasks (e.g., "clean a plate and put it in the cupboard").
  • WebShop: A simulated e-commerce website where agents must navigate, search, and select products based on complex user requirements.

They tested these across two frozen model backbones: Qwen2.5-14B-Instruct and Qwen3.5-27B.

Key Insights and Ideas

  • Memory Influence is Concentrated: The authors found that most self-extracted memories are "inert"—they barely shift the agent's decision-making process at all. Only a small tail of candidates produces substantial changes (see Figure 2 in the paper).
  • Direction Matters: A large shift in probability isn't always good. A memory can sometimes suppress a correct action or reinforce a wrong one. The Aspecter component is critical because it filters for "role-aligned" changes, ensuring the memory actually helps the agent do the right thing.
  • Co-Evolution: As the agent accumulates these verified memories, its behavior changes, which in turn leads to the extraction of even more refined memories. This creates a feedback loop the authors call "co-evolution," allowing a frozen model to improve its performance over time.

Key Results

Anchor consistently outperformed strong baselines like Reflexion, ExpeL, and AdaMEM while being more resource-efficient.

  • Performance Gains: On ALFWorld, Anchor reached 96.27% success (using Qwen3.5-27B), a significant jump from the strongest baseline's 92.54%. On WebShop, it improved mean scores by up to 17.2% relative to baselines.
  • Efficiency: Anchor is remarkably lean. It retained 41% fewer memory cards than unfiltered retention on ALFWorld.
  • Cost-Effectiveness: On WebShop, Anchor's total token usage was lower than all four major cost-comparison baselines, including being ~98.5% cheaper than AdaMEM in terms of token consumption.

Limitations and Future Work

  • Probability Access: Anchor requires access to the model's token log-probabilities (logprobs) to measure distribution shifts. This makes it difficult to use with "black-box" closed-source APIs that don't provide this data.
  • Action Space Constraints: The current implementation works best in environments with explicit, discrete candidate actions. Applying this to open-ended generation (like creative writing) would require new ways to measure "behavioral shift."
  • Future Directions: The authors suggest future work could involve calibrating these measures for broader tasks and exploring ways to implement Anchor using model-generated scores or rankings when logprobs are unavailable.

Summary updated

Abstract

LLM agents have made impressive progress in coding and mathematical reasoning, motivating growing interest in learning from long-horizon interaction. Memory supports this process, but plausible lessons can be redundant or reinforce inappropriate actions. Can a memory's source decision provide a practical test for whether to retain it? We localize consequential trajectory steps, extract candidate memories, and replay each source decision with and without its candidate under the same context. This reveals two properties: most candidates barely shift the probed distribution, and both successful and failed source actions can be reinforced or suppressed. These observations motivate Anchor, which tests shift magnitude and source-role alignment before retention. Admitted memories guide later trajectories, enabling experience-driven improvement under a frozen policy. On held-out ALFWorld and WebShop tasks, Anchor improves success rate and mean score by up to 19.9% and 17.2%, respectively, relative to the strongest baseline for each backbone. With Qwen3.5-27B, it retains 41% fewer ALFWorld cards than unfiltered retention and outperforms all four WebShop cost-comparison baselines while using fewer tokens.
BibTeX
@misc{yin2026from,
  author = {Xinxiang Yin and Xinglang Zhang and Haodong Chen and Zhiqing Cui and Weidong Tang and Yuxuan Wang and Junrui Wang and Jingyi Zhang and Jianfeng Zhang and Jian Qian},
  title = {{From Reflection to Co-Evolution: Self-Evolving Agents with Locally Verified Memory}},
  year = {2026},
  eprint = {2611.10382},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.10382},
}
Figure