TL;DR
ColaBench is a new benchmark designed to evaluate how personal AI agents use long-term interaction history to acquire and apply external knowledge. Along with the benchmark, the authors introduce MemKB, a retrieval method that uses state summaries and explicit memory-document links to significantly improve accuracy, especially in complex tasks requiring information from multiple historical sessions.
What Problem Does This Paper Address?
Imagine asking a personal agent, "Am I eligible for a caregiver grant?" To answer correctly, the agent doesn't just need to find the grant's rules in a database; it needs to remember that three months ago you mentioned living alone, and last week you mentioned caring for a relative with Alzheimer’s.
Current research often treats "memory" (recalling what the user said) and "knowledge acquisition" (finding external info) as separate problems. However, in real-world scenarios, these are deeply intertwined: your personal history should dictate what knowledge the agent searches for and how it applies those rules to your specific situation. This paper addresses the lack of benchmarks that jointly evaluate these two challenges, ensuring that agents don't just provide "correct" information that is ultimately "unsuitable" for the user.
Methods
The authors tackle this problem from two sides: a new evaluation framework and a novel retrieval architecture.
1. ColaBench Construction
The researchers built ColaBench using a Plan–Create–Review multi-agent framework. The core innovation here is the Evidence Graph, which traces the dependencies between personal facts disclosed in conversation, the documents the agent has seen, and the final requirements for a correct answer.
2. MemKB (Memory-Linked Retrieval)
To solve the tasks in ColaBench, the authors propose MemKB. Unlike standard RAG (Retrieval-Augmented Generation) which might just search a knowledge base based on the current query, MemKB does three things:
- State Summaries: It maintains a "state bank" of the user's circumstances (e.g., "lives alone," "takes prescription pain meds") to inform the very first search query.
- Memory-Document Links: It creates explicit links between specific memories and the documents that were read when those memories were formed.
- Hybrid Fusion: It uses Reciprocal Rank Fusion (RRF) to combine results from three channels: standard semantic search, keyword search, and the new memory-linked channel.
Datasets
The authors introduced ColaBench, a synthetic benchmark focused on health and daily living.
- Scope: 10 detailed user Profiles and 349 complex tasks.
- Knowledge Base: 13,033 documents sourced from MedlinePlus, openFDA, the National Cancer Institute, and nursing textbooks.
- Complexity: It includes "multi-Fact" tasks where the agent must synthesize information from multiple different historical interactions to reach the correct conclusion.
Key Insights and Ideas
- History as a Retrieval Compass: The paper demonstrates that user history shouldn't just be "context" for the LLM; it should actively guide the retrieval engine to find documents that the agent wouldn't otherwise know it needs.
- The Multi-Fact Challenge: Agents struggle most when evidence is scattered across time. MemKB showed its greatest strengths here, suggesting that explicit state tracking is vital for long-term coherence.
- The "Personal Assumption" Trap: A fascinating finding in the error analysis was that agents often fail by making leaps—such as assuming a user is taking an opioid just because they mentioned "prescription pain medicine." This highlights a need for agents to maintain "conditional uncertainty" when history is ambiguous.
Key Results
- Higher Accuracy: MemKB achieved 86.25% accuracy, outperforming the strong Mem0+Hybrid baseline (79.94%).
- Efficiency: MemKB reached these answers in fewer interaction rounds (4.07 vs 4.44), meaning it finds the right information faster.
- Complexity Gains: On tasks requiring the integration of multiple historical facts, MemKB saw a massive 14.41 percentage point gain over the baseline.
Limitations and Future Work
- Synthetic Nature: While grounded in real medical documents, the user interactions are synthetic. The authors note that generalization to real-world, messy human dialogue remains to be fully tested.
- Domain Specificity: The benchmark is currently focused on health and daily living. Expanding this to other domains (like legal or financial advice) is a logical next step.
- Clinical Validation: The authors explicitly state that the health-related answers generated during testing have not been clinically validated and should not be used for medical advice.
- Dynamic Environments: The current knowledge base is "frozen." Future work could explore how agents handle rapidly changing information (like news or live web content) alongside long-term memory.
Summary updated