MemPO: Self-Memory Policy Optimization for Long-Horizon Agents

Ruoran Li, Xinghua Zhang, Haiyang Yu, Shitong Duan, Xiang Li, Wenxin Xiang, Chonghua Liao, Xudong Guo, Yongbin Li, Jinli Suo

TL;DR

MemPO (Self-Memory Policy Optimization) is a novel reinforcement learning framework that trains LLM agents to autonomously manage, summarize, and update their own interaction history. By treating memory management as an intrinsic policy action and optimizing it using a task-aware reward system, MemPO slashes token consumption by up to 73% while significantly outperforming traditional full-context and RAG-based agent baselines on complex, long-horizon tasks.

What Problem Does This Paper Address?

When large language model (LLM) agents tackle complex, multi-step problems—such as deep research or data analysis—they typically rely on the ReAct paradigm. In this setup, the environment's feedback from each step is appended to the interaction history and fed back into the prompt.

However, this causes the context window to grow linearly with every turn, leading to three major bottlenecks:

  1. Context Limits: Agents quickly hit the hard token ceiling of the underlying LLM.
  2. Exorbitant Costs: Long contexts drastically increase token usage and inference latency.
  3. Performance Degradation: LLMs suffer from the "lost in the middle" phenomenon, where their reasoning abilities falter when forced to process massive prompts.

While existing solutions use Retrieval-Augmented Generation (RAG) or external databases to store and look up past interactions, these methods are passive. They rely purely on embedding similarity and are completely decoupled from the agent's actual task objectives. The agent cannot proactively decide what information is worth keeping and what should be discarded to solve the problem at hand.

Methods

To turn memory management into an active, learned capability, the authors formalize the agent's action space into three distinct tags: <mem>, <think>, and <tool_call>.

Instead of feeding the entire history into the LLM, the agent uses a truncated context: it only receives the previous step's memory and environment response. It must then use the <mem> tag to synthesize a new, cumulative summary before moving on to reasoning (<think>) and taking action (<tool_call>).

The self-memory inference process using truncated context

To train this capability without a massive computational overhead, the authors build on Group Relative Policy Optimization (GRPO) and introduce a dual-advantage credit assignment mechanism:

  • Trajectory-Level Advantage ($A^T$): A sparse, global reward assigned to all tokens in a trajectory based on whether the final answer is correct and properly formatted.
  • Memory-Level Advantage ($A^M$): A dense, step-level reward specifically designed to evaluate the quality of the <mem> block. It measures the conditional probability of the correct answer given the generated memory content ($P[s^{ans} | \tau_i(s^{mem}_t)]$). If the agent writes a high-quality summary that retains crucial task-relevant details, the model's internal confidence in generating the correct final answer increases.
Overview of the MemPO training and reward pipeline

Datasets

The authors evaluated MemPO across a diverse suite of benchmarks:

  • Multi-Objective Tasks: Synthesized by combining queries from HotpotQA and Natural Questions (NQ). The tasks were scaled progressively from 2 to 10 objectives to simulate highly complex, long-horizon scenarios.
  • Single-Objective Multi-Hop QA: Evaluated on standard benchmarks including 2WikiMultiHopQA, Bamboogle, Musique, TriviaQA, and PopQA.
  • Deep Research Tasks (Out-of-Domain): Tested on GAIA, Frames, and WebWalkerQA to evaluate how well the self-memory policy generalizes to real-world web search environments.

Key Insights and Ideas

  • Memory as an Intrinsic Policy: Rather than treating memory as an engineering problem (like setting up a vector database), MemPO treats it as a machine learning problem. Memory optimization is jointly trained alongside reasoning and tool use.
  • Confidence as a Proxy for Information Quality: The core "aha moment" of the paper is using the model's own posterior probability of the correct answer to score its memory. This provides a fine-grained, dense reward signal that effectively solves the credit assignment problem in multi-turn interactions.
  • Robustness to Context Truncation: As shown in the authors' ablation studies, MemPO-trained models remain remarkably stable whether they are given a 1-step history, a 3-step history, or the full context. This proves that the agent genuinely learns to compress all vital information into the <mem> block.

Key Results

MemPO achieved state-of-the-art (SOTA) performance across the board, demonstrating that agents can be both smarter and cheaper:

  • Higher Accuracy: On the grueling 10-objective task, MemPO achieved an absolute F1 score gain of 25.98 over the base model and 7.1 over the previous SOTA baseline (MEM1).
  • Massive Token Savings: Compared to full-context baselines, MemPO reduced total token usage by 67.58% and slashed peak token usage per step by 73.12%.
Ablation studies showing performance gains and context strategy stability
  • Strong Generalization: Despite being trained on local Wikipedia dumps, MemPO generalized seamlessly to live web search environments, outperforming or matching specialized deep research baselines on GAIA and Frames.

Limitations and Future Work

While powerful, the authors note a key limitation in their advantage estimation: because tool invocations vary wildly at different steps, the memory states across different rollout trajectories are not completely equivalent. This can introduce a slight bias when calculating group-based advantages. Although the authors introduced a bias-correction term ($\epsilon$) to alleviate this, they note that more refined solutions will be necessary for highly complex, open-ended environments. Future work will focus on scaling this self-memory approach to broader, more diverse real-world applications.

Summary updated

Abstract

Long-horizon agents face the challenge of growing context size during interaction with environment, which degrades the performance and stability. Existing methods typically introduce the external memory module and look up the relevant information from the stored memory, which prevents the model itself from proactively managing its memory content and aligning with the agent's overarching task objectives. To address these limitations, we propose the self-memory policy optimization algorithm (MemPO), which enables the agent (policy model) to autonomously summarize and manage their memory during interaction with environment. By improving the credit assignment mechanism based on memory effectiveness, the policy model can selectively retain crucial information, significantly reducing token consumption while preserving task performance. Extensive experiments and analyses confirm that MemPO achieves absolute F1 score gains of 25.98 over the base model and 7.1 over the previous SOTA baseline, while reducing token usage by 67.58% and 73.12%. The code is released at https://github.com/TheNewBeeKing/MemPO.
BibTeX
@inproceedings{li2026mempo,
  author = {Ruoran Li and Xinghua Zhang and Haiyang Yu and Shitong Duan and Xiang Li and Wenxin Xiang and Chonghua Liao and Xudong Guo and Yongbin Li and Jinli Suo},
  title = {{MemPO: Self-Memory Policy Optimization for Long-Horizon Agents}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1166},
}

Loading related content...

Figure