Remembering Has a Price: Learning Memory Worth under Resource Constraints

Guannan Lai, Zi-Yang Bai, Jun-Peng Jiang, Han-Jia YearXiv 2026 · arXiv

TL;DR

Lifelong learning agents often struggle when operating under tight resource budgets because they waste precious tokens on "relevant" but ultimately unhelpful memories. This paper introduces TokenWorth, a lightweight framework that learns to estimate the actual utility—or "worth"—of a memory from sparse execution feedback, allowing agents to achieve higher task success rates while significantly reducing token consumption.

What Problem Does This Paper Address?

In the ideal world of research, lifelong agents have infinite context windows and unlimited interaction rounds. In the real world, agents face stringent constraints: API costs, hardware latency, and limited memory capacity.

The authors identify a "fragility" in existing lifelong-memory methods (like RAG or MemRL) when these resources are jointly constrained. The core issue is that semantic relevance is a trap. Just because a past experience is similar to a current query doesn't mean it will help the agent solve the task. Under a tight budget, retrieving a relevant-looking but ineffective memory consumes scarce context and can actually hinder execution. The challenge shifts from "how do we find similar experiences?" to "which experiences are actually worth the cost of remembering?"

Figure 1: Left: Representative local results with Qwen2.5-14B-Instruct under the evaluated stringent joint resource constraints; TokenWorth remains effective where several memory baselines do not. Right: Resource-constrained lifelong agents operate under bounded generation, interaction, and memory budgets.

Methods

The authors propose TokenWorth, a framework that manages the entire memory lifecycle—retrieval, admission, and eviction—based on a learned "worth" score.

The Scoring Architecture

TokenWorth decomposes memory worth into two parts:

  1. Linear Base Scorer: Captures stable, interpretable global patterns using features like textual similarity, historical utility, reliability, and recency.
  2. RFM Residual Learner: Uses a Recursive Feature Machine (RFM) to model non-linear, query-specific corrections that the base scorer might miss.

Two-Timescale Adaptation

To stay stable yet adaptive, the system uses two different learning speeds:

  • Fast Adaptation: Uses Kernel Ridge Regression (KRR) to rapidly update the residual learner based on recent execution outcomes.
  • Slow Adaptation: Uses Average Gradient Outer Product (AGOP) to evolve the underlying "feature geometry," essentially learning which aspects of a query-memory pair are most predictive of success over the long term.
Overview of the TokenWorth workflow

Datasets

The framework was evaluated across several heterogeneous environments:

  • LifelongAgentBench: Includes Database interaction (DB), Operating System tasks (OS), and Knowledge Graph reasoning (KG).
  • Generalization Benchmarks: Online lifelong variants of BigCodeBench (coding) and ALFWorld (embodied decision-making).

Evaluations were conducted using both local models (Qwen2.5-14B-Instruct) and API-based models (DeepSeek-V4-Flash).

Key Insights and Ideas

  • "Remembering Has a Price": Every token used to provide context is a token taken away from the agent's ability to reason or act. In resource-constrained settings, memory management is a zero-sum game.
  • Utility > Relevance: The paper argues that we should stop optimizing for retrieval similarity and start optimizing for downstream task success.
  • Efficiency as a Byproduct of Quality: Interestingly, by selecting higher-value memories, the agent often solves tasks in fewer steps. TokenWorth doesn't just work within a budget; it actively saves resources.
  • Lightweight is Key: Unlike methods that require fine-tuning the backbone LLM or complex "reflection" steps (which cost tokens), TokenWorth's controller is a small, external module that requires no additional LLM calls for its internal logic.

Key Results

TokenWorth consistently outperformed sophisticated baselines (including RAG, MemRL, and Reflexion) across all tested environments.

  • Higher Success Rates: On the DB task using a local backbone, TokenWorth achieved a 65.0% success rate, more than doubling the performance of the strongest baseline (RAG at 33.0%).
  • Superior Efficiency: In the OS environment, TokenWorth reduced output token usage by over 50% compared to a standard no-memory agent, while still maintaining a higher success rate.
  • Consistency: The advantage held true across both local and API-based deployments, suggesting the method is robust to different backbone architectures.

Limitations and Future Work

The authors acknowledge a few key hurdles:

  • Causal Attribution: Feedback is only observed for the memories that are actually selected. This makes it difficult to know if an unselected memory would have performed even better (the "counterfactual" problem).
  • Credit Assignment: When a task fails, it's hard to distinguish if the failure was due to a "bad" memory or simply a mistake by the frozen LLM backbone.
  • Future Directions: The authors suggest that developing more budget-efficient ways to handle credit assignment—determining exactly which part of a trajectory led to success or failure—is a critical next step for the field.

Summary updated

Abstract

Lifelong agents continually accumulate and reuse experience to improve after deployment, yet in practice they often operate under tight generation, interaction, and memory budgets. Under such stringent joint constraints, we find that existing lifelong-memory methods can become ineffective, with several sophisticated approaches offering little advantage over simple retrieval or even no-memory agents. This reveals a more fundamental problem: resource scarcity makes each memory decision more consequential. When only a small amount of past experience can be retained or reused and opportunities for reasoning and recovery are limited, retrieving a relevant but ineffective memory can consume scarce context and hinder downstream execution. The key challenge is therefore not simply to retrieve relevant experience, but to determine which experiences are actually worth retaining and reusing. To address this challenge, we propose TokenWorth, a lightweight online memory framework that learns query-dependent memory worth from sparse execution feedback and applies it consistently to memory retrieval and bounded memory management. Across three LifelongAgentBench environments and both local and API-based deployments, TokenWorth achieves the highest task success rate while using the fewest output tokens among the evaluated methods. These results suggest that effective lifelong adaptation under resource constraints depends not on remembering more, but on learning what is worth remembering. Our code is publicly available at https://github.com/AIGNLAI/TokenWorth.
BibTeX
@misc{lai2026remembering,
  author = {Guannan Lai and Zi-Yang Bai and Jun-Peng Jiang and Han-Jia Ye},
  title = {{Remembering Has a Price: Learning Memory Worth under Resource Constraints}},
  year = {2026},
  eprint = {2610.26453},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.26453},
}
Figure