TL;DR
BioMem presents the first systematic investigation into "agentic memory"—persistent, evolving information systems—for biomedical LLMs. The authors identify critical failure modes in how models retrieve and utilize medical data and introduce HMRU (Hybrid Memory Retrieval and Utilization), a training-free framework that improves biomedical reasoning by 8.6% and closes nearly half the performance gap between memory-augmented systems and full-context models.
What Problem Does This Paper Address?
While Retrieval-Augmented Generation (RAG) has become a standard for grounding LLMs in external knowledge, biomedical applications require more than just static lookups. Agents in this domain must handle agentic memory: the ability to store, update, and reuse information across long-term patient interactions and multi-session clinical reasoning.
However, we don't yet understand the reliability of these persistent memory systems in a high-stakes medical context. This paper addresses three core questions:
- Fidelity: Does the memory accurately reflect the original source after multiple updates?
- Retrieval vs. Utilization: If a model retrieves the correct "answer-bearing" memory, does it actually use it to produce the right answer?
- Task Sensitivity: How do factors like retrieval depth and class imbalance affect clinical predictions compared to simple knowledge retrieval?
Methods
The researchers first benchmarked five existing memory strategies (including A-MEM, Mem0, and LightMem) across four state-of-the-art LLMs. Finding significant gaps in how these models handled biomedical nuances, they proposed HMRU (Hybrid Memory Retrieval and Utilization).
HMRU is a training-free framework consisting of three main components:
- Hybrid Retrieval: Combines dense vector retrieval (semantic) with BM25 (lexical) to ensure that specific medical terminology isn't missed.
- Cross-Encoder Reranking: Refines the retrieved set to prioritize the most relevant evidence.
- Task-Aware Utilization: Uses specialized prompt instructions to guide the model on how to weigh retrieved evidence, emphasizing entity-relation consistency for reasoning and outcome prevalence for clinical predictions.
Datasets
The study evaluates memory performance across three distinct settings:
- Biomedical Knowledge Reasoning: Using RiTeK-PharmKG and RiTeK-ADInt, which require reasoning over complex medical knowledge graphs.
- Clinical Prediction: Utilizing MIMIC-III (Mortality and Readmission), TCGA-Reports (Cancer classification), and MIMIC-IV OOD (Opioid overdose prediction).
- Long-term Conversational Memory: Using the medical subset of RealMem, which tests temporal reasoning and information updates across multiple sessions.
Key Insights and Ideas
-
The Utilization Gap: Simply retrieving the right information is not enough. The authors found that even when "answer-bearing" memories were present in the prompt, models often failed to utilize them correctly. Evidence-to-outcome flows showing that retrieving relevant evidence does not guarantee correct utilization.
-
Memory Drift: As agentic memory systems "evolve" (summarizing and rewriting notes over time), they tend to introduce source-unsupported information. In some tasks, up to 44% of memory nodes contained "hallucinated" or unsupported context after several rewrites.
- Similarity $\neq$ Outcome: In clinical prediction, retrieving a "semantically similar" patient record doesn't necessarily help predict a future outcome. This highlights a fundamental limitation of standard RAG for prospective risk assessment.
- History Length is a Red Herring: Contrary to expectations, longer conversation history didn't consistently degrade performance. Instead, the accessibility and integration of specific evidence were the primary bottlenecks.
Key Results
HMRU demonstrated significant improvements over baseline agentic memory systems:
- Overall Gains: Achieved an average relative improvement of 8.6% across QA and conversational benchmarks.
- Error Reduction: Reduced conversational answer errors by 24.9%.
- Closing the Gap: Closed 48.0% of the performance gap between standard memory systems and "Full-Context" inference (where the model sees the entire history at once).
- Mechanistic Success: The "Task-Aware" component was particularly effective at reducing errors involving multiple entities and inappropriate "yes/no" responses. Mechanistic analysis showing how HMRU components reduce retrieval and utilization failures.
Limitations and Future Work
The authors note that while HMRU excels at knowledge retrieval and reasoning, clinical outcome prediction remains a significant challenge. Standard retrieval objectives (finding similar text) are often insufficient for predicting risks like opioid overdose.
Future work should focus on outcome-aware retrieval—systems that don't just look for similar patients, but specifically look for patients with similar trajectories and outcomes. Additionally, addressing the "fidelity drift" in evolving memories is crucial for maintaining the long-term safety of biomedical agents.
Summary updated