MemRL: Self-Evolving Agents via Runtime Reinforcement Learning on Episodic Memory

Shengtao Zhang, Jiaqian Wang, Ruiwen Zhou, Junwei Liao, Yuchen Feng, Zhuo Li, Yujie Zheng, Weinan Zhang, Ying Wen, Zhiyu li, Feiyu Xiong, Yutao Qi, Bo Tang, Muning Wen

Abstract

The hallmark of human intelligence is the self-evolving ability to master new skills by learning from past experiences. However, current AI agents struggle to emulate this self-evolution: fine-tuning is computationally expensive and prone to catastrophic forgetting, while existing memory-based methods rely on passive semantic matching that often retrieves noise. To address these challenges, we propose MemRL, a non-parametric approach that evolves via reinforcement learning on episodic memory. By decoupling stable reasoning from plastic memory, MemRL employs a Two-Phase Retrieval mechanism to filter noise and identify high-utility strategies through environmental feedback. Extensive experiments on HLE, BigCodeBench, ALFWorld, and Lifelong Agent Bench demonstrate that MemRL significantly outperforms state-of-the-art baselines, confirming that MemRL effectively reconciles the stability-plasticity dilemma, enabling continuous runtime improvement without weight updates. Code is available at https://github.com/MemTensor/MemRL.

Introduction: The Stability-Plasticity Dilemma

Current AI agents face a fundamental challenge known as the "stability-plasticity dilemma." To adapt to new tasks after deployment, agents typically rely on two methods, both of which have significant drawbacks. Fine-tuning modifies internal weights, which is computationally intensive and risks "catastrophic forgetting" of prior knowledge. Alternatively, standard Retrieval-Augmented Generation (RAG) is passive; it retrieves past experiences based on semantic similarity—how much a past problem looks like the current one—without considering whether those past attempts were actually successful. This often results in the agent retrieving "noise" or "distractor" memories that lead to repeated failures.

MemRL architecture inspired by human cognitive processes like constructive episodic simulation.

MemRL addresses this by decoupling stable reasoning (a frozen LLM backbone) from plastic episodic memory. Inspired by human cognitive processes like constructive episodic simulation, MemRL allows agents to "self-evolve" at runtime by learning from environmental feedback without altering the underlying model weights.

Methodology: Reinforcement Learning on Episodic Memory

The core of MemRL is a non-parametric approach that treats memory retrieval as a value-based decision process. The framework is built upon three primary components:

1. Intent-Experience-Utility Triplets

Unlike traditional RAG which stores simple key-value pairs, MemRL organizes memory into triplets:

  • Intent: The user's query or the agent's goal.
  • Experience: The specific solution trajectory or action sequence taken.
  • Utility: A learned scalar value (Q-value) representing the historical effectiveness of that experience for the given intent.

2. Two-Phase Retrieval

To overcome the limitations of semantic-only matching, MemRL implements a two-stage process:

  • Phase A (Similarity Recall): The system first identifies a pool of semantically similar experiences from the memory bank, acting as a broad filter.
  • Phase B (Value-Aware Selection): The agent re-ranks these candidates based on their learned utility scores. This critical step allows the agent to ignore "near-misses"—strategies that appear relevant but have a history of failure—and prioritize high-utility solutions.

3. Utility-Driven Updates

Learning in MemRL occurs through environmental feedback. When an agent completes a task, it receives a reward signal. Using a Monte Carlo-style update, the system adjusts the utility scores of the retrieved memories. Success increases the utility, while failure decreases it, allowing the agent's "wisdom" to evolve continuously.

The end-to-end MemRL framework featuring two-phase retrieval and utility updates.

Experimental Evaluation

The researchers evaluated MemRL across four diverse benchmarks: BigCodeBench (complex coding), ALFWorld (embodied navigation), Lifelong Agent Bench (OS and DB management), and Humanity’s Last Exam (HLE) (high-level reasoning).

Key Findings

  • Performance Gains: MemRL consistently outperformed state-of-the-art baselines like MemP and standard RAG, achieving an average increase of +3.8% in Cumulative Success Rate (CSR). In exploration-heavy environments like ALFWorld, the gain reached +6.2%.
  • Utility over Similarity: The results confirm that semantic similarity is a poor proxy for usefulness. MemRL's ability to prioritize successful strategies over "similar" failures is its primary driver of success.
  • Model-Agnostic Wisdom: In a notable experiment, a memory bank trained by a powerful model (Gemini-3-pro) was used by a smaller model (Qwen3-235B). The smaller model's performance tripled, demonstrating that episodic memory can effectively transfer "wisdom" across different architectures.
  • Efficiency and Robustness: MemRL introduces negligible algorithmic overhead, as updates are simple scalar operations. Furthermore, it showed high resistance to interference when merging memory banks from different domains, thanks to the Phase A semantic gating.

Conclusion and Future Directions

MemRL provides a robust solution to the stability-plasticity dilemma by enabling runtime learning through an external, evolving memory bank. While highly effective, future work will address challenges such as precise credit assignment in multi-memory tasks—potentially using Shapley values—and mitigating "reward hacking" from noisy feedback. By keeping the reasoning engine stable and the memory plastic, MemRL paves the way for AI agents that truly learn and grow from their own history.

Summary updated

Abstract

The hallmark of human intelligence is the self-evolving ability to master new skills by learning from past experiences. However, current AI agents struggle to emulate this self-evolution: fine-tuning is computationally expensive and can introduce destabilizing weight updates, increasing the risk of catastrophic forgetting, while existing memory-based methods rely on passive semantic matching that often retrieves noise. To address these challenges, we propose MemRL, a non-parametric approach that evolves via reinforcement learning on episodic memory. By decoupling stable reasoning from plastic memory, MemRL employs a Two-Phase Retrieval mechanism to filter noise and identify high-utility strategies through environmental feedback. Extensive experiments on Humanity's Last Exam, BigCodeBench, ALFWorld, and Lifelong Agent Bench demonstrate that MemRL significantly outperforms state-of-the-art baselines, suggesting that MemRL offers a practical way to balance stability and plasticity during runtime improvement.
BibTeX
@misc{zhang2026memrl,
  author = {Shengtao Zhang and Jiaqian Wang and Ruiwen Zhou and Junwei Liao and Yuchen Feng and Zhuo Li and Yujie Zheng and Weinan Zhang and Ying Wen and Zhiyu li and Feiyu Xiong and Yutao Qi and Bo Tang and Muning Wen},
  title = {{MemRL: Self-Evolving Agents via Runtime Reinforcement Learning on Episodic Memory}},
  year = {2026},
  eprint = {2601.03192},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2601.03192},
}
Figure