Abstract
The hallmark of human intelligence is the self-evolving ability to master new skills by learning from past experiences. However, current AI agents struggle to emulate this self-evolution: fine-tuning is computationally expensive and prone to catastrophic forgetting, while existing memory-based methods rely on passive semantic matching that often retrieves noise. To address these challenges, we propose MemRL, a non-parametric approach that evolves via reinforcement learning on episodic memory. By decoupling stable reasoning from plastic memory, MemRL employs a Two-Phase Retrieval mechanism to filter noise and identify high-utility strategies through environmental feedback. Extensive experiments on HLE, BigCodeBench, ALFWorld, and Lifelong Agent Bench demonstrate that MemRL significantly outperforms state-of-the-art baselines, confirming that MemRL effectively reconciles the stability-plasticity dilemma, enabling continuous runtime improvement without weight updates. Code is available at https://github.com/MemTensor/MemRL.
Introduction: The Stability-Plasticity Dilemma
Current AI agents face a fundamental challenge known as the "stability-plasticity dilemma." To adapt to new tasks after deployment, agents typically rely on two methods, both of which have significant drawbacks. Fine-tuning modifies internal weights, which is computationally intensive and risks "catastrophic forgetting" of prior knowledge. Alternatively, standard Retrieval-Augmented Generation (RAG) is passive; it retrieves past experiences based on semantic similarity—how much a past problem looks like the current one—without considering whether those past attempts were actually successful. This often results in the agent retrieving "noise" or "distractor" memories that lead to repeated failures.
MemRL addresses this by decoupling stable reasoning (a frozen LLM backbone) from plastic episodic memory. Inspired by human cognitive processes like constructive episodic simulation, MemRL allows agents to "self-evolve" at runtime by learning from environmental feedback without altering the underlying model weights.
Methodology: Reinforcement Learning on Episodic Memory
The core of MemRL is a non-parametric approach that treats memory retrieval as a value-based decision process. The framework is built upon three primary components:
1. Intent-Experience-Utility Triplets
Unlike traditional RAG which stores simple key-value pairs, MemRL organizes memory into triplets:
- Intent: The user's query or the agent's goal.
- Experience: The specific solution trajectory or action sequence taken.
- Utility: A learned scalar value (Q-value) representing the historical effectiveness of that experience for the given intent.
2. Two-Phase Retrieval
To overcome the limitations of semantic-only matching, MemRL implements a two-stage process:
- Phase A (Similarity Recall): The system first identifies a pool of semantically similar experiences from the memory bank, acting as a broad filter.
- Phase B (Value-Aware Selection): The agent re-ranks these candidates based on their learned utility scores. This critical step allows the agent to ignore "near-misses"—strategies that appear relevant but have a history of failure—and prioritize high-utility solutions.
3. Utility-Driven Updates
Learning in MemRL occurs through environmental feedback. When an agent completes a task, it receives a reward signal. Using a Monte Carlo-style update, the system adjusts the utility scores of the retrieved memories. Success increases the utility, while failure decreases it, allowing the agent's "wisdom" to evolve continuously.
Experimental Evaluation
The researchers evaluated MemRL across four diverse benchmarks: BigCodeBench (complex coding), ALFWorld (embodied navigation), Lifelong Agent Bench (OS and DB management), and Humanity’s Last Exam (HLE) (high-level reasoning).
Key Findings
- Performance Gains: MemRL consistently outperformed state-of-the-art baselines like MemP and standard RAG, achieving an average increase of +3.8% in Cumulative Success Rate (CSR). In exploration-heavy environments like ALFWorld, the gain reached +6.2%.
- Utility over Similarity: The results confirm that semantic similarity is a poor proxy for usefulness. MemRL's ability to prioritize successful strategies over "similar" failures is its primary driver of success.
- Model-Agnostic Wisdom: In a notable experiment, a memory bank trained by a powerful model (Gemini-3-pro) was used by a smaller model (Qwen3-235B). The smaller model's performance tripled, demonstrating that episodic memory can effectively transfer "wisdom" across different architectures.
- Efficiency and Robustness: MemRL introduces negligible algorithmic overhead, as updates are simple scalar operations. Furthermore, it showed high resistance to interference when merging memory banks from different domains, thanks to the Phase A semantic gating.
Conclusion and Future Directions
MemRL provides a robust solution to the stability-plasticity dilemma by enabling runtime learning through an external, evolving memory bank. While highly effective, future work will address challenges such as precise credit assignment in multi-memory tasks—potentially using Shapley values—and mitigating "reward hacking" from noisy feedback. By keeping the reasoning engine stable and the memory plastic, MemRL paves the way for AI agents that truly learn and grow from their own history.
Summary updated