Facts Fade Fast: Evaluating Memorization of Outdated Medical Knowledge in Large Language Models

Juraj Vladika, Mahdi Dhaini, Florian MatthesEMNLP 2025

TL;DR

Large Language Models (LLMs) show great promise in healthcare, but their reliance on static training data means they can quickly become outdated, providing potentially harmful medical advice. This paper introduces new datasets to evaluate this "outdated knowledge" problem and finds that current LLMs consistently rely on older information, even when newer evidence exists.

What Problem Does This Paper Address?

The rapid evolution of medical knowledge poses a significant challenge for AI systems designed to assist in healthcare. LLMs are trained on vast datasets, and while this allows them to "memorize" a lot of information, it also means they can retain outdated medical facts and recommendations. This can lead to incorrect health advice or flawed clinical reasoning, a serious safety concern. The paper aims to quantify how much outdated medical knowledge LLMs retain and to understand the factors contributing to this phenomenon.

Methods

The researchers developed two novel question-answering (QA) datasets derived from systematic reviews (SLRs) to evaluate LLMs' medical knowledge.

  • MedRevQA: A large dataset of 16,501 QA pairs covering general biomedical knowledge from SLRs.
  • MedChangeQA: A smaller subset of 512 QA pairs specifically designed to capture instances where the medical consensus has changed over time, allowing for a direct comparison of LLM performance on older vs. newer information.

They then evaluated eight prominent LLMs on these datasets using a zero-shot setting, meaning the models were asked to answer questions based solely on their internal knowledge without any additional context. The performance was measured using precision, recall, and F1 scores. They also conducted an analysis to trace the source of outdated knowledge and explored potential mitigation strategies.

Datasets

  • MedRevQA: A newly created dataset of 16,501 QA pairs derived from medical systematic reviews.
  • MedChangeQA: A subset of 512 QA pairs from MedRevQA where the conclusion of the systematic review has changed over time, indicating an evolution in medical consensus.
  • PubMed: Used as the source for Cochrane systematic review abstracts to construct the datasets.
  • Cochrane Collaboration: The primary source for systematic reviews used in dataset construction.

Key Insights and Ideas

  • Consistent Reliance on Outdated Knowledge: All eight LLMs evaluated showed a consistent tendency to rely on outdated medical information, even when newer, contradictory evidence was available.
  • Performance Decline with Recency: The average F1 scores of LLMs on questions from MedRevQA generally declined for questions originating from more recent years, suggesting a stronger ability to recall older information.
  • Memorization as a Root Cause: The paper posits that the LLMs' ability to recall specific facts from their training data, often referred to as memorization, is a primary driver of this outdated knowledge problem.
  • Temporal Decay of Medical Knowledge is Underexplored: While memorization in general domains has been studied, the specific issue of temporal decay in medical knowledge has received less attention.
  • Potential for Harm: Providing outdated medical advice can have serious safety implications for patients and healthcare professionals.

Key Results

  • Outdated Knowledge is Pervasive: Across all tested LLMs, performance on the MedChangeQA dataset was consistently lower when using the latest labels as ground truth compared to using outdated labels. This indicates that LLMs are better at recalling older, superseded information.
  • Specific Model Performance: Mistral 24B showed the best overall performance on the full MedRevQA dataset, indicating a strong grasp of the medical knowledge landscape. However, even this model struggled with recent information. Llama 3.3 showed the most significant improvement when evaluated against the latest labels, suggesting it retained more recent knowledge compared to others.
  • Citations Reveal Age: Analysis of LLM answers revealed a tendency to cite older studies and systematic reviews, with terms like "Cochrane" and "systematic review" appearing frequently, often referring to decade-old publications.
  • RAG Shows Promise: A simple retrieval-augmented generation (RAG) approach, where the top PubMed abstract was added as context, showed improvements in F1 scores, suggesting that external knowledge retrieval can help mitigate the outdated knowledge problem.

Limitations and Future Work

  • Semi-Automatic Dataset Construction: MedRevQA was constructed semi-automatically, which might introduce some imperfections in generated questions and labels. Manual analysis of 100 instances showed high accuracy (92-95%), but it's noted that human annotation is also imperfect.
  • LLM Availability and Cost: Not all relevant LLMs could be tested due to availability or computational cost.
  • Lack of Expert Human Evaluation: The study did not involve in-depth human evaluation by medical experts, which could provide a more rigorous assessment.
  • Future Work: The authors suggest several directions:
  • Advanced RAG Techniques: Developing more sophisticated RAG methods, including filtering, re-ranking, and semantic search.
  • Nuanced Labels and Long-Form Answers: Exploring more detailed labeling schemes and generating longer, more explanatory answers.
  • Mitigation Strategies: Investigating knowledge editing, machine unlearning, and continual learning to address outdated knowledge.
  • Broader Evaluation: Expanding the evaluation to include more LLMs and a wider range of medical domains.

Summary updated

BibTeX
@inproceedings{vladika2025facts,
  author = {Juraj Vladika and Mahdi Dhaini and Florian Matthes},
  title = {{Facts Fade Fast: Evaluating Memorization of Outdated Medical Knowledge in Large Language Models}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure