TL;DR
MEMENTO is a context-management framework that enables small LLM agents (like Qwen3-8B) to process massive documents by learning when and how to "compact" their history into notes. By combining supervised bootstrapping with reinforcement learning, MEMENTO achieves performance comparable to models 100x its size at a fraction of the cost, effectively managing books up to 2.5 million tokens while staying within a 20K token context window.
What Problem Does This Paper Address?
Large Language Model (LLM) agents are increasingly used for "long-horizon" tasks, such as answering complex questions about entire books. However, they face a fundamental "context window" bottleneck. Even as windows grow to 1M+ tokens, models often suffer from "lost in the middle" phenomena or performance degradation as the context fills up.
The standard solution is compaction: the agent summarizes its previous steps into a "note" and clears its history to make room for new information. The catch? Off-the-shelf models are surprisingly bad at this. They either write useless notes, forget to take notes entirely, or fail to follow the formatting required to make the notes useful for future steps. Reinforcement Learning (RL) usually fails to fix this because the model is so unlikely to generate a "perfect" note-taking trajectory by chance that there is nothing for the RL algorithm to reward.
Methods
The authors developed a two-stage training pipeline to "teach" an 8B-parameter model (Qwen3) how to manage its own memory:
- Synthetic SFT (Supervised Fine-Tuning): To solve the "cold start" problem, they used a larger teacher model (GLM-5.3) to generate successful task rollouts. They then retrospectively injected high-quality notes into these trajectories and fine-tuned the 8B model on this data. This "bootstraps" the basic behavior of note-taking.
- RL with GRPO: They refined the model using Group Relative Policy Optimization (GRPO). The reward function wasn't just based on the final answer; it included "process signals" like note validity (frequency, placement, and diversity) and confidence calibration.
At inference time, MEMENTO uses a "harness" that supports three distinct strategies (shown in Figure 1):
- Compaction: Starting a fresh window with only the saved notes.
- Context Folding: Keeping notes plus the most recent turns to maintain "local" momentum.
- Confidence-based Backtracking: If the agent gives an answer with low verbalized confidence, it "rewinds" to an earlier note and tries a different search path.
Datasets
The researchers focused on Book-level Multi-hop Question Answering (Book QA), where answers are buried across hundreds of pages. They used:
- LiteraryQA: Narrative-heavy questions.
- NovelHopQA: Requires connecting multiple pieces of evidence.
- SagaScale (English subset): Extremely long-context benchmarks.
- NovelQA: Used as a held-out evaluation set to ensure the model wasn't just memorizing training books.
Key Insights and Ideas
- Co-design is King: You cannot train the model in a vacuum. The training must be "co-designed" with the test-time harness. Because MEMENTO was trained to expect its context to be reset and replaced by notes, it naturally generalized to more advanced techniques like "folding" and "backtracking" without further training.
- SFT as a Catalyst: Figure 4 shows that without the initial SFT bootstrapping, RL quickly "gives up" on note-taking because it's too hard. SFT provides the "right strategy," and RL then makes that strategy "reliable."
- Confidence as a Search Signal: By rewarding the model for being "calibrated" (knowing when it's likely wrong), the authors enabled a search-like behavior at test time. If the model says "I'm not sure," the system automatically triggers a backtrack.
Key Results
MEMENTO (8B) punches significantly above its weight class:
- Accuracy: It raised Qwen3-8B's accuracy from 36.9% to 67.8%.
- Efficiency: It nearly matched the performance of GLM-5.3 (a ~750B model) while using 1/9th the cost and half the tokens per rollout.
- Vs. Proprietary Giants: It outperformed a GPT-5.5 agent (13.1% in the same harness) and even beat GPT-5.5 using its full native context window (56.7%), all while being 54x cheaper.
- Consistency: MEMENTO never held more than 20K tokens in context at once, proving that smart management beats raw window size.
Limitations and Future Work
- Overconfidence: Late in RL training, the model tends to become overconfident, even when wrong. This reduces the effectiveness of backtracking, as the model doesn't "know" it needs to retry.
- Heuristic-based Rewards: The "note validity" reward relies on hand-crafted heuristics (e.g., "don't take two notes in a row"). Future work could involve a more semantic "meta-judge" to reward notes based on their actual utility for solving the task.
- Generalization: The study was limited to English Book QA. Whether these note-taking strategies transfer to coding or web research remains an open question.
Summary updated