TL;DR
Lifelong learning agents often struggle when operating under tight resource budgets because they waste precious tokens on "relevant" but ultimately unhelpful memories. This paper introduces TokenWorth, a lightweight framework that learns to estimate the actual utility—or "worth"—of a memory from sparse execution feedback, allowing agents to achieve higher task success rates while significantly reducing token consumption.
What Problem Does This Paper Address?
In the ideal world of research, lifelong agents have infinite context windows and unlimited interaction rounds. In the real world, agents face stringent constraints: API costs, hardware latency, and limited memory capacity.
The authors identify a "fragility" in existing lifelong-memory methods (like RAG or MemRL) when these resources are jointly constrained. The core issue is that semantic relevance is a trap. Just because a past experience is similar to a current query doesn't mean it will help the agent solve the task. Under a tight budget, retrieving a relevant-looking but ineffective memory consumes scarce context and can actually hinder execution. The challenge shifts from "how do we find similar experiences?" to "which experiences are actually worth the cost of remembering?"
Methods
The authors propose TokenWorth, a framework that manages the entire memory lifecycle—retrieval, admission, and eviction—based on a learned "worth" score.
The Scoring Architecture
TokenWorth decomposes memory worth into two parts:
- Linear Base Scorer: Captures stable, interpretable global patterns using features like textual similarity, historical utility, reliability, and recency.
- RFM Residual Learner: Uses a Recursive Feature Machine (RFM) to model non-linear, query-specific corrections that the base scorer might miss.
Two-Timescale Adaptation
To stay stable yet adaptive, the system uses two different learning speeds:
- Fast Adaptation: Uses Kernel Ridge Regression (KRR) to rapidly update the residual learner based on recent execution outcomes.
- Slow Adaptation: Uses Average Gradient Outer Product (AGOP) to evolve the underlying "feature geometry," essentially learning which aspects of a query-memory pair are most predictive of success over the long term.
Datasets
The framework was evaluated across several heterogeneous environments:
- LifelongAgentBench: Includes Database interaction (DB), Operating System tasks (OS), and Knowledge Graph reasoning (KG).
- Generalization Benchmarks: Online lifelong variants of BigCodeBench (coding) and ALFWorld (embodied decision-making).
Evaluations were conducted using both local models (Qwen2.5-14B-Instruct) and API-based models (DeepSeek-V4-Flash).
Key Insights and Ideas
- "Remembering Has a Price": Every token used to provide context is a token taken away from the agent's ability to reason or act. In resource-constrained settings, memory management is a zero-sum game.
- Utility > Relevance: The paper argues that we should stop optimizing for retrieval similarity and start optimizing for downstream task success.
- Efficiency as a Byproduct of Quality: Interestingly, by selecting higher-value memories, the agent often solves tasks in fewer steps. TokenWorth doesn't just work within a budget; it actively saves resources.
- Lightweight is Key: Unlike methods that require fine-tuning the backbone LLM or complex "reflection" steps (which cost tokens), TokenWorth's controller is a small, external module that requires no additional LLM calls for its internal logic.
Key Results
TokenWorth consistently outperformed sophisticated baselines (including RAG, MemRL, and Reflexion) across all tested environments.
- Higher Success Rates: On the DB task using a local backbone, TokenWorth achieved a 65.0% success rate, more than doubling the performance of the strongest baseline (RAG at 33.0%).
- Superior Efficiency: In the OS environment, TokenWorth reduced output token usage by over 50% compared to a standard no-memory agent, while still maintaining a higher success rate.
- Consistency: The advantage held true across both local and API-based deployments, suggesting the method is robust to different backbone architectures.
Limitations and Future Work
The authors acknowledge a few key hurdles:
- Causal Attribution: Feedback is only observed for the memories that are actually selected. This makes it difficult to know if an unselected memory would have performed even better (the "counterfactual" problem).
- Credit Assignment: When a task fails, it's hard to distinguish if the failure was due to a "bad" memory or simply a mistake by the frozen LLM backbone.
- Future Directions: The authors suggest that developing more budget-efficient ways to handle credit assignment—determining exactly which part of a trajectory led to success or failure—is a critical next step for the field.
Summary updated