ColaBench: Evaluating Memory-Guided Knowledge Acquisition in Personal Agents

Wenjie Zhang, Zhichao Shi, Xuhui Jiang, Xiaojun Wu, Cehao Yang, Chengjin Xu, Xiaoning SongarXiv 2026 · arXiv

TL;DR

ColaBench is a new benchmark designed to evaluate how personal AI agents use long-term interaction history to acquire and apply external knowledge. Along with the benchmark, the authors introduce MemKB, a retrieval method that uses state summaries and explicit memory-document links to significantly improve accuracy, especially in complex tasks requiring information from multiple historical sessions.

What Problem Does This Paper Address?

Imagine asking a personal agent, "Am I eligible for a caregiver grant?" To answer correctly, the agent doesn't just need to find the grant's rules in a database; it needs to remember that three months ago you mentioned living alone, and last week you mentioned caring for a relative with Alzheimer’s.

Current research often treats "memory" (recalling what the user said) and "knowledge acquisition" (finding external info) as separate problems. However, in real-world scenarios, these are deeply intertwined: your personal history should dictate what knowledge the agent searches for and how it applies those rules to your specific situation. This paper addresses the lack of benchmarks that jointly evaluate these two challenges, ensuring that agents don't just provide "correct" information that is ultimately "unsuitable" for the user.

Methods

The authors tackle this problem from two sides: a new evaluation framework and a novel retrieval architecture.

1. ColaBench Construction

The researchers built ColaBench using a Plan–Create–Review multi-agent framework. The core innovation here is the Evidence Graph, which traces the dependencies between personal facts disclosed in conversation, the documents the agent has seen, and the final requirements for a correct answer.

Figure 1: Plan–Create–Review construction: (a) document collection; (b) grounded Profiles and Topics; (c) public interaction; (d) history-dependent Item construction; and (e) Item enhancement and independent validation. The growing Evidence Graph guides knowledge exploration and constrains generation and review through recorded Fact disclosures, document exposure, and supporting evidence.

2. MemKB (Memory-Linked Retrieval)

To solve the tasks in ColaBench, the authors propose MemKB. Unlike standard RAG (Retrieval-Augmented Generation) which might just search a knowledge base based on the current query, MemKB does three things:

  • State Summaries: It maintains a "state bank" of the user's circumstances (e.g., "lives alone," "takes prescription pain meds") to inform the very first search query.
  • Memory-Document Links: It creates explicit links between specific memories and the documents that were read when those memories were formed.
  • Hybrid Fusion: It uses Reciprocal Rank Fusion (RRF) to combine results from three channels: standard semantic search, keyword search, and the new memory-linked channel.
Figure 2: The MemKB architecture, showing how offline construction of memory cards informs online retrieval.

Datasets

The authors introduced ColaBench, a synthetic benchmark focused on health and daily living.

  • Scope: 10 detailed user Profiles and 349 complex tasks.
  • Knowledge Base: 13,033 documents sourced from MedlinePlus, openFDA, the National Cancer Institute, and nursing textbooks.
  • Complexity: It includes "multi-Fact" tasks where the agent must synthesize information from multiple different historical interactions to reach the correct conclusion.

Key Insights and Ideas

  • History as a Retrieval Compass: The paper demonstrates that user history shouldn't just be "context" for the LLM; it should actively guide the retrieval engine to find documents that the agent wouldn't otherwise know it needs.
  • The Multi-Fact Challenge: Agents struggle most when evidence is scattered across time. MemKB showed its greatest strengths here, suggesting that explicit state tracking is vital for long-term coherence.
  • The "Personal Assumption" Trap: A fascinating finding in the error analysis was that agents often fail by making leaps—such as assuming a user is taking an opioid just because they mentioned "prescription pain medicine." This highlights a need for agents to maintain "conditional uncertainty" when history is ambiguous.

Key Results

  • Higher Accuracy: MemKB achieved 86.25% accuracy, outperforming the strong Mem0+Hybrid baseline (79.94%).
  • Efficiency: MemKB reached these answers in fewer interaction rounds (4.07 vs 4.44), meaning it finds the right information faster.
  • Complexity Gains: On tasks requiring the integration of multiple historical facts, MemKB saw a massive 14.41 percentage point gain over the baseline.
Figure 6: Accuracy breakdown by fact count and history pairs, showing MemKB's superior performance on complex tasks.

Limitations and Future Work

  • Synthetic Nature: While grounded in real medical documents, the user interactions are synthetic. The authors note that generalization to real-world, messy human dialogue remains to be fully tested.
  • Domain Specificity: The benchmark is currently focused on health and daily living. Expanding this to other domains (like legal or financial advice) is a logical next step.
  • Clinical Validation: The authors explicitly state that the health-related answers generated during testing have not been clinically validated and should not be used for medical advice.
  • Dynamic Environments: The current knowledge base is "frozen." Future work could explore how agents handle rapidly changing information (like news or live web content) alongside long-term memory.

Summary updated

Abstract

Personal agents need to draw on long-term interaction history to retrieve and apply external knowledge suited to a user's current request. Even an answer grounded in relevant documents can be unsuitable when it overlooks the user's circumstances. This creates two connected challenges: using historical conditions to identify the knowledge needed for a new request, and applying that knowledge to the user's situation. Existing benchmarks advance memory recall, personalized retrieval, and memory-guided action, while the joint evaluation of these two challenges remains underexplored. We introduce ColaBench, a synthetic benchmark in health and daily living, with 349 tasks across 10 Profiles and a knowledge base of 13,033 documents. Its Evidence Graph connects personal conditions to their disclosures across public interaction sessions, records prior document exposure, and links answer requirements to personal and newly acquired document evidence. We also introduce MemKB, which uses state summaries to inform queries and rule application, and memory-document links to retrieve knowledge associated with the user's history. MemKB achieves 86.25% Accuracy with an average of 4.07 interaction rounds per task. Task-level analysis finds larger gains on tasks integrating information across multiple historical Facts; error analysis further identifies unsupported personal assumptions and rule misapplication as obstacles to using retrieved evidence. ColaBench provides traceable tasks and an evaluation framework for studying how memory guides knowledge acquisition and rule application.
BibTeX
@misc{zhang2026colabench,
  author = {Wenjie Zhang and Zhichao Shi and Xuhui Jiang and Xiaojun Wu and Cehao Yang and Chengjin Xu and Xiaoning Song},
  title = {{ColaBench: Evaluating Memory-Guided Knowledge Acquisition in Personal Agents}},
  year = {2026},
  eprint = {2610.30989},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.30989},
}
Figure