Persistent Experiential Memory for Language Agents

107 concepts89 papers6 proposals

When humans interact with complex software, browse the web, or solve domain-specific problems, we rarely approach each new task from first principles. If a specific sequence of actions bypasses a broken user interface, or if a particular debugging strategy resolves a recurring compiler error, we internalize that routine. We use our past successes and failures to build an implicit library of workflows and strategic principles.

In contrast, standard Large Language Model (LLM) agents suffer from what can be described as operational amnesia or episode isolation. As Sumers et al. formalize in their cognitive architecture framework, language agents typically rely on short-term context windows or external document retrieval, but they generally lack a mechanism to autonomously update their core procedural behaviors based on long-term experience. Because their underlying model weights remain frozen during inference, they treat every new task or session as a completely blank slate (Oliveira et al.; Tan et al.). This statelessness forces agents to redundantly re-calculate basic subroutines, repeat previously solved errors, and fail at tasks requiring iterative trial-and-error (Qian et al.; Wang et al.).

To bridge this experience accumulation gap, a new paradigm has emerged: Persistent Experiential Memory. Instead of modifying a model's weights through expensive fine-tuning, these frameworks introduce a closed-loop lifecycle where an agent interacts with an environment, reflects on its performance, distills raw trajectories into structured knowledge, and stores this knowledge in a cross-session memory layer to guide future tasks.

[Task Execution] ──> [Outcome Evaluation] ──> [Abstraction & Distillation]
       ▲                                                    │
       │                                                    ▼
[Contextual Retrieval] <─────────────────────────── [Persistent Memory Store]

Abstracting Experiences into Workflows and Skills

Storing raw, unedited interaction logs is highly inefficient due to context window constraints, redundancy, and noise (Xia et al.). Therefore, the core technical challenge lies in abstraction—translating low-level execution steps into high-level, reusable routines.

One prominent approach is to induce reusable workflows. For example, Wang et al. developed Agent Workflow Memory (AWM), a framework that automatically abstracts successful action sequences from past experiences into structured routines. When facing a task in an unseen domain, the agent references these workflows rather than planning from scratch.

Similarly, Xiang et al. introduce FlowSearcher, which synthesizes memory-guided workflows specifically for web information seeking, ensuring that effective search paths are preserved across episodes. To make these behaviors fully adaptive, Xia et al. propose SkillRL, which recursively distills raw interaction data into a library of discrete skills that evolve alongside the agent's execution policy.


Contextual Replay and Analogical Reasoning

How does an agent determine which past experience is relevant to a novel problem? The answer lies in analogical reasoning and dynamic retrieval.

The foundational intuition of analogical reasoning in LLMs is demonstrated by Yu et al. with Thought Propagation, where models are prompted to explore and solve analogous problems first, then propagate those insights to the target task. To scale this dynamically across complex environments, Yuan et al. developed Retrieval-Augmented Planning (RAP), which retrieves relevant past trajectories from a contextual memory to guide multimodal robotic and text-based agents.

Rather than relying on static prompts, Liu et al. propose Contextual Experience Replay (CER). CER distills raw environmental interactions into a dynamic memory of environment dynamics and reusable skills, enabling agents to achieve performance gains entirely at inference time without any parameter updates. Similarly, Liu et al. introduce WebCoach, which utilizes a dedicated "Coach" module to retrieve and inject cross-session advice from past failures and successes directly into the agent's prompt.


Policy-Level Reflection and Strategic Principles

Beyond individual workflows, agents can distill their histories into high-level strategic principles or "tips" that govern behavior across broad domains.

Wu et al. introduce EvolveR, a framework where agents autonomously distill their own interaction histories into a library of "strategic principles," utilizing reinforcement learning to evolve their decision-making policies over time. For competitive or imperfect-information environments, Zhang et al. propose Agent-Pro, which shifts the focus from immediate action-correction to policy-level reflection, optimizing the agent's behavioral prompts via depth-first search.

This hierarchical abstraction is highly effective in practical applications:

  • Mobile Assistants: Wang et al. demonstrate this with Mobile-Agent-E, which maintains a long-term memory of natural language "Tips" and executable "Shortcuts" to allow smartphone assistants to self-evolve across multiple applications.
  • Test-Time Scaling: Ouyang et al. introduce ReasoningBank, which pairs a self-updating reasoning memory with test-time scaling. This architecture creates a virtuous cycle: allocating more compute allows the agent to discover better strategies, which are then distilled into memory to guide future exploration more efficiently.

Why It Works: Non-Parametric Policy Adaptation

The fundamental reason persistent experiential memory works is that it shifts the agent from a static text predictor to a dynamic, non-parametric learning system.

Standard prompting forces an LLM to rely entirely on the general knowledge encoded in its pre-trained weights. When an agent encounters a failure, that failure is lost the moment the context window resets. By externalizing experience into a structured, queryable memory layer, these frameworks decouple learning from weight updates.

The memory layer acts as a non-parametric policy modifier. It dynamically reshapes the agent's local action distribution by conditioning the LLM on historical evidence of what worked and what failed. As a result, the agent effectively "fine-tunes" its behavior at test time, achieving self-evolution without the computational burden or catastrophic forgetting associated with traditional gradient-based training.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_static_policy_limitation_0fdc4621,
  title = {static policy limitation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/static-policy-limitation/rdr_0fdc462198db422bb88aaf42df5c437b},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_static_policy_limitation_0fdc4621_v1,
  title = {static policy limitation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/static-policy-limitation/rdr_0fdc462198db422bb88aaf42df5c437b/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake