Memory-Augmented Large Language Model-Based Agent with Cross-Task Experience Learning

Sijia Wang, Gengzhi Zhang, Xuefeng Chen, Liang FengOpenReview 2026 · Submission to ICLR 2026 · Rejected · OpenReview

TL;DR

This paper introduces a memory-augmented LLM agent that learns not just from its own mistakes, but from experiences gained across different tasks. By combining a static "source" memory of transferable knowledge with a dynamic "target" memory for task-specific adaptation, and balancing them with a clever retrieval mechanism, the agent achieves significantly higher data efficiency and success rates on complex, multi-turn benchmarks like WebShop.

What Problem Does This Paper Address?

While Large Language Model (LLM) agents are great at following instructions, they often behave like they have no long-term memory. Most existing agents treat every interaction as an isolated event or, at best, only remember what happened in the current session.

This leads to two major bottlenecks:

  1. Low Data Efficiency: Agents often need to fail many times in a new environment before they "get it," because they can't leverage similar experiences from other domains.
  2. Static Retrieval: Current memory-augmented agents usually use rigid rules to look up past experiences, making them unable to adapt their strategy as they become more proficient at a specific task.

The authors aim to bridge this gap by allowing agents to "transfer" knowledge from one task (e.g., buying clothes) to another (e.g., buying electronics) while dynamically shifting their focus from general advice to specific experience as they learn.

Methods

The researchers proposed a framework consisting of three main components:

  • Dual-Memory Architecture:
    • Source Experience Memory ($M_s$): A static library of successful "high-utility" experiences from related but distinct tasks. Think of this as the agent's "textbook."
    • Target Experience Memory ($M_t$): A dynamic store that accumulates experiences from the current task in real-time. This is the agent's "notebook."
  • Dynamic Memory Retrieval Mechanism (DMRM): Instead of just grabbing the most similar text, the agent ranks potential memories using cosine similarity of sentence embeddings. Crucially, the retrieval is length-aware; it fills the LLM's context window with as many relevant experiences as possible without overflowing.
  • Adaptive Source Control: This is the "aha" feature. The agent uses a suppression threshold ($n$). Once the agent has successfully completed a task a certain number of times (populating $M_t$), it stops looking at the source memory ($M_s$). This prevents "negative transfer"—where old, general knowledge might actually distract the agent from specific nuances of the current task.
  • Reinforcement Learning (RL) Integration: The target memory isn't just a list; it’s refined using Q-learning to estimate the long-term utility of specific state-action pairs.

Datasets

The authors evaluated their agent on WebShop, a large-scale benchmark for online shopping agents.

  • Scale: Over 1 million product descriptions and 12,000 human-written instructions.
  • Complexity: Tasks are multi-turn and require navigating a web interface (searching, clicking, selecting attributes) to find a product that matches a specific description.
  • Domains: The study focused on four specific domains: Fashion, Beauty, Electronics, and Grocery.

Key Insights and Ideas

  • The "Goldilocks" Threshold: The ablation studies found that a suppression threshold of $n=2$ was optimal. If the agent switches to task-specific memory too early ($n=1$), performance drops. If it waits too long ($n=4$), the "noise" from the general source memory starts to degrade performance.
  • Cross-Domain Transfer Works: Even when the source and target domains are semantically distant (e.g., using "Fashion" experience to help with "Grocery" shopping), the agent still sees a performance boost. This suggests that the strategies for navigation and decision-making are transferable, even if the products are different.
  • Structured Templates Matter: The agent doesn't just store raw logs; it converts experiences into a structured "Encouraged/Discouraged" natural language format. This makes it much easier for the frozen LLM to "reason" about why a past action was good or bad.

Key Results

The proposed method was compared against several baselines, including ReAct (a popular reasoning/acting framework) and REMEMBERER (a state-of-the-art memory agent).

  • Higher Success Rates: The "Mem+Transfer" agent achieved a 40% success rate, outperforming REMEMBERER (38%) and ReAct (36%).
  • Better Alignment: On the "Average Score" metric (which measures how well the selected product matches the user's intent on a scale of 0 to 1), the proposed agent scored 0.70, the highest among all tested methods.
  • Generalization: The agent showed robust performance when tested on "unseen" domains, proving that the cross-task learning mechanism effectively captures generalizable shopping strategies.

Limitations and Future Work

  • Static Source Memory: Currently, the source memory is pre-constructed and static. The authors suggest that in the future, the source memory could be updated dynamically as the agent masters more tasks.
  • Similarity Metric Reliance: The system relies heavily on sentence embeddings for retrieval. In environments where semantic similarity doesn't perfectly correlate with task utility, the retrieval might fail.
  • Computational Overhead: Managing and searching through two separate memory stores, especially as they grow, could introduce latency in real-time applications.
  • Manual Templates: The transformation of experiences into structured text relies on predefined templates, which might not scale to every possible type of agent task.

Summary updated

Abstract

Large Language Model (LLM)-based agents have demonstrated impressive capabilities in complex decision-making and multi-turn instruction-following tasks. To enhance knowledge retention and contextual adaptability, recent work has equipped these agents with memory modules that store and reuse historical interaction experiences. However, existing memory-augmented approaches face two key limitations: they often require large amounts of interaction data during early training to reach competitive performance, resulting in low data efficiency; and they rely on static, self-derived experience reuse strategies, limiting their ability to adapt when prior learning is insufficient and preventing the use of transferable knowledge from related tasks. Building on these observations, in this paper, we propose a memory-augmented LLM agent with cross-task experience learning, designed to improve data efficiency and adaptability. Our method augments the conventional task-specific memory with an additional source experience memory that retains transferable knowledge from related but distinct tasks. We further introduce a dynamic memory retrieval mechanism that adaptively draws from both task and source memories, allowing the agent to balance prior task-specific experiences with cross-task knowledge according to the current context and progression. We validate the proposed method on the WebShop benchmark, which comprises diverse, multi-turn instruction-following tasks across product domains with varying semantic complexity. Experimental results show that our approach consistently outperforms state-of-the-art memory-augmented LLM agents in task success rate and generalization, demonstrating the effectiveness of the proposed memory architecture and retrieval mechanism.
BibTeX
@misc{wang2026memory,
  author = {Sijia Wang and Gengzhi Zhang and Xuefeng Chen and Liang Feng},
  title = {{Memory-Augmented Large Language Model-Based Agent with Cross-Task Experience Learning}},
  year = {2026},
  url = {https://openreview.net/forum?id=0GMt2OWeCb},
  howpublished = {OpenReview},
  note = {Submitted to ICLR 2026},
}
Figure