Rethinking Continual Experience Internalization for Self-Evolving LLM Agents

Jingwen Chen, Wenkai Yang, Shengda Fan, Wenbo Nie, Chenxing Sun, Shaodong Zheng, Yangen Hu, Lu Pan, Ke Zeng, Yankai LinarXiv 2026 · arXiv · OpenReview

TL;DR

This paper tackles the problem of "capability collapse" in LLM agents that learn from their own experiences over multiple iterations. They propose a three-pronged approach—principle-level experience, step-wise injection, and off-policy distillation—to achieve stable, self-evolving LLM agents that continuously improve rather than degrade.

What Problem Does This Paper Address?

Large Language Models (LLMs) are increasingly being developed as agents that can interact with environments, learn from their experiences, and improve over time. A key mechanism for this is "experience internalization," where past interactions are converted into reusable parametric capabilities within the model. While this has shown promise for single-step learning, the paper identifies a critical issue: when LLM agents learn from their own evolving experiences over multiple iterations (self-evolution), existing internalization methods suffer from "capability collapse." Instead of compounding improvements, their performance degrades. This paper aims to understand why this collapse happens and proposes a more robust method for stable, multi-iteration experience internalization.

Methods

The authors systematically investigate three key dimensions of experience internalization:

  • Experience Granularity: They compare "instance-level" experience (raw details from a specific interaction) with "principle-level" experience (abstracted strategies, rules, and failure patterns).
  • Experience Injection Pattern: They contrast "global injection" (where experience is treated as a fixed context for the entire trajectory) with "step-wise injection" (where experience is dynamically selected based on the current decision state).
  • Internalization Regime: They analyze the difference between "on-policy context-distillation" (where the student model generates trajectories, and the teacher corrects them) and "off-policy context-distillation" (where the teacher generates high-quality trajectories, and the student learns from them).

By analyzing these dimensions, they propose a "recipe" for stable experience internalization.

Datasets

The paper uses the following datasets for training and evaluation:

  • Training Corpus: A 15K-example corpus from five public web-reasoning QA datasets: WebWalkerQA-silver, DeepDive, WebShaper, WebDancer, and SailorFog-QA.
  • Evaluation Benchmarks:
    • WebWalkerQA (in-domain)
    • GAIA-Text-103 (out-of-domain)
    • BrowseComp-ZH (out-of-domain)

Key Insights and Ideas

  • Principle over Instance: Abstracting general strategies (principle-level experience) is far more effective for long-term learning than retaining specific interaction details (instance-level experience). Instance-level details are too brittle and don't generalize well to new situations.
  • Dynamic is Better: Injecting experience dynamically based on the current decision state (step-wise injection) is crucial, especially for long-horizon tasks. Global injection, which treats experience as static context, can lead to misalignment and premature decision-making.
  • Teacher Quality Matters: Off-policy distillation, where a high-quality teacher generates trajectories, provides a much more stable training signal than on-policy distillation. On-policy methods are limited by the student's own potentially flawed states, leading to a feedback loop of errors.
  • Sustainability is Key: The core contribution is demonstrating how to achieve sustained improvement across multiple self-evolution iterations, moving beyond single-iteration gains.

Key Results

  • Experience Granularity: Principle-level experience consistently outperformed instance-level experience across iterations, showing more stable performance gains. For example, on WebWalkerQA, principle-level experience led to a success rate of 29.0% in Iteration 1, which was sustained at 30.0% by Iteration 3, whereas instance-level experience dropped from 22.8% to 19.7%.
  • Experience Injection Pattern: Step-wise injection significantly outperformed global injection. On WebWalkerQA, step-wise injection achieved 29.0% in Iteration 1 and maintained 30.0% in Iteration 3, while global injection dropped from 23.2% to 16.8% over the same period. Table 1 also showed step-wise injection improving performance from 23.2% to 31.2% on WebWalkerQA when using Qwen self-generated experience.
  • Internalization Regime: Off-policy context-distillation consistently yielded better and more stable results than on-policy context-distillation across all benchmarks and models. For instance, on Qwen3-4B with GAIA, off-policy distillation achieved 22.7% in Iteration 1 and 24.0% in Iteration 3, while on-policy distillation dropped from 22.8% to 19.4%.
  • Combined Approach: The proposed combination of principle-level experience, step-wise injection, and off-policy distillation demonstrated robust performance gains across self-evolution iterations, outperforming the base model and maintaining the ability to leverage explicit experience. Figure 6 shows sustained internalized performance and in-context responsiveness.

Limitations and Future Work

  • Domain Specificity: The experiments were primarily conducted on web-reasoning agent tasks. Further evaluation is needed to assess if these findings generalize to other domains, languages, and agent settings.
  • Other Factors: The paper explored three key dimensions. Other factors like experience-pool size, selector quality, and filtering criteria might also influence stability and warrant further investigation.
  • Broader Impact Concerns: While the work aims to build more reliable agents, the authors acknowledge that stable internalization could also reinforce undesirable behaviors (e.g., incorrect, biased, or unsafe patterns) if the accumulated experience is flawed. They emphasize the need for trajectory filtering, experience-pool auditing, human oversight, and safeguards in high-risk settings. Future work could explore these aspects more comprehensively.

Summary updated

Abstract

Experience internalization converts contextual experience from past interactions into reusable parametric capability, offering a promising path toward continual learning in large language models (LLMs). While prior work has predominantly focused on single-iteration transfer, we discover that under multi-iteration experience learning, existing methods suffer from a progressive capability collapse rather than compounding improvement. We systematically examine this failure through three vital dimensions of experience internalization: (1) Experience Granularity: We find that principle-level experience is more durable than instance-level experience, as it effectively abstracts transferable strategies away from trajectory-specific details. (2) Experience Injection Pattern: Our analysis reveals that step-wise injection significantly outperforms global injection by aligning experience with intermediate decision states, a property that is critical for long-horizon tool use. (3) Internalization Regime: We demonstrate that off-policy context-distillation on high-quality teacher trajectories provides a substantially more stable training signal than on-policy context-distillation, which is inherently limited by local corrections on student-induced flawed states. Together, these insights yield a simple yet robust recipe for stable and sustainable experience internalization, providing concrete guidance for engineering self-evolving and continually learning LLMs.
BibTeX
@misc{chen2026rethinking,
  author = {Jingwen Chen and Wenkai Yang and Shengda Fan and Wenbo Nie and Chenxing Sun and Shaodong Zheng and Yangen Hu and Lu Pan and Ke Zeng and Yankai Lin},
  title = {{Rethinking Continual Experience Internalization for Self-Evolving LLM Agents}},
  year = {2026},
  eprint = {2606.04703},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2606.04703},
}

Loading related content...

Figure