Abstract
Large Language Models (LLMs) are typically stateless, losing the benefits of interaction once the context window is cleared. While implicit memory (model weights) can be updated via fine-tuning, it is computationally expensive and opaque. Conversely, explicit memory (external text) is accessible but lacks a formal optimization framework. This paper introduces $EM^2$ (Explicit Memory Learning with Expectation-Maximization), a framework that treats textual memory as a learnable latent variable. By applying the Expectation-Maximization algorithm, $EM^2$ allows LLMs to iteratively refine their external memory based on performance, enabling "stateless" models to improve through experience without weight updates.
Introduction
A significant limitation of current LLMs is their inability to learn from ongoing interactions. Traditional approaches rely on either updating internal weights (implicit memory) or utilizing external databases (explicit memory). Implicit memory updates are resource-intensive and often result in "black box" behaviors. Explicit memory, such as Retrieval-Augmented Generation (RAG) or scratchpads, is more transparent but lacks a principled mechanism for self-improvement. $EM^2$ bridges this gap by formalizing the update process for textual memory using the Expectation-Maximization (EM) algorithm.
Methodology: The $EM^2$ Framework
The core innovation of $EM^2$ is treating non-differentiable textual memory as a latent variable. The framework operates in a two-step iterative cycle:
The E-Step (Expectation)
In the E-step, the model utilizes its current explicit memory—consisting of strategies, demonstrations, or past solutions—to generate predictions for new tasks. The system calculates the expected success of these predictions based on the stored information.
The M-Step (Maximization)
In the M-step, the model evaluates its performance on a representative validation set, which is a diverse cluster of recent problems identified through clustering techniques like KMeans or DBSCAN. The memory is then updated by selecting the strategies that maximize the likelihood of correct outcomes.
To maintain memory quality, $EM^2$ employs an uncertainty threshold. Only high-confidence predictions, determined by normalized entropy, are integrated into the memory. This prevents the model from learning from its own hallucinations or errors.
Experimental Setup and Datasets
The $EM^2$ framework was rigorously tested across 14 datasets spanning three domains:
- Word Math Problems (WMP): Including GSM8K, MATH, and SVAMP.
- Commonsense QA: Including StrategyQA and ARC-Challenge.
- Symbolic Understanding: Tasks from the Big-Bench benchmark, such as Object Counting and Date Understanding.
Key Insights
- Text as a Latent Variable: By framing text as a hidden state, $EM^2$ applies classical statistical optimization to modern LLM prompting.
- Model-Agnostic Portability: Explicit memory generated by a larger model (e.g., LLaMA-3-70B) can be effectively utilized by smaller models (e.g., LLaMA-3-8B), facilitating knowledge transfer.
- Entropy-Based Filtering: Using entropy as a filter ensures the model only "remembers" reliable information, mitigating the risk of error propagation.
- Directional Memory Updates: Unlike simple buffers, $EM^2$ provides a mathematically grounded direction for updates, ensuring statistical improvement over time.
Results and Discussion
Experimental results demonstrate that $EM^2$ significantly enhances model performance. On the GSM8K benchmark, the framework achieved a 5.83% improvement over Zero-Shot Chain-of-Thought (CoT) and a 3.02% gain over standard CoT. The benefits were more pronounced in larger, more capable models like Qwen-2 and LLaMA-3-70B.
Furthermore, $EM^2$ is computationally efficient compared to fine-tuning, as the M-step does not require gradient calculations. The framework also showed robustness across various embedding models and clustering algorithms.
Limitations and Future Directions
A primary constraint of $EM^2$ is the requirement for "white-box" access to the model's internal probability distributions to calculate entropy and perplexity. This limits its applicability to closed-source APIs. Future research could explore adapting the framework for open-ended generative tasks, utilizing more complex memory structures like knowledge graphs, and incorporating human-in-the-loop feedback to further refine the maximization process.
Summary updated