Can We Edit LLMs for Long-Tail Biomedical Knowledge?

Xinhao Yi, Jake Lever, Kevin Bryson, Zaiqiao Meng

Abstract

Large Language Models (LLMs) are increasingly integrated into clinical workflows, yet their reliance on training frequency creates a significant performance gap between "popular" and "long-tail" (rare) biomedical knowledge. Knowledge editing (KE) has been proposed as a solution to inject specific facts into models without exhaustive retraining. This study investigates the efficacy of KE in the biomedical domain using a new benchmark, CliKT (Clinical Knowledge Triples). Our findings reveal that while KE significantly boosts performance on rare facts, a persistent gap remains compared to popular knowledge. We identify the high prevalence of "one-to-many" relationships—common in medicine but rare in general domains—as a primary obstacle to effective knowledge editing for the long tail.

Introduction

In the medical field, the difference between common and rare knowledge can be a matter of life and death. While an LLM might easily recall the treatment for Type 1 Diabetes (popular knowledge), it often fails when queried about rare conditions like Evans Syndrome (long-tail knowledge). This discrepancy arises because LLMs are fundamentally frequency-biased, prioritizing patterns that appear often in their massive training corpora.

Knowledge editing (KE) techniques, such as ROME, MEMIT, and MEND, offer a promising alternative to full-model fine-tuning by allowing targeted updates to specific model weights. However, the clinical reliability of these methods remains unproven. This research asks: Can current editing techniques effectively bridge the performance gap for rare biomedical information?

Methodology and the CliKT Benchmark

To evaluate this, the researchers developed the CliKT (Clinical Knowledge Triples) benchmark. CliKT consists of over 100,000 triples extracted from SNOMED CT and validated against the PubMed corpus (37 million abstracts). By calculating the co-occurrence number—the frequency with which a subject and object appear together in the same document—the team categorized knowledge into bins ranging from rare to popular.

Overview of the study workflow, illustrating the transition from long-tail knowledge identification to knowledge editing and result analysis.

The study employed two phases:

  1. Knowledge Probing: Assessing the base model's inherent knowledge.
  2. Knowledge Editing: Applying five state-of-the-art methods (ROME, MEMIT, MEND, IKE, and FT) to inject or correct specific biomedical facts.

Key Results: The Persistent Gap

The initial probing phase confirmed a stark "Frequency Trap." For instance, BioMedLM's accuracy on long-tail knowledge was 22.86% lower than its performance on popular facts before any intervention.

Applying knowledge editing (specifically the ROME method) yielded a substantial improvement, increasing BioMedLM's accuracy on long-tail facts by approximately 52%. However, the "long-tail gap" did not disappear. Even after editing, performance on rare knowledge remained 16.15% lower than that of edited popular knowledge. This suggests that the model's underlying architecture or the editing process itself struggles more with rare associations than with common ones.

The "One-to-Many" Hurdle

The most significant finding of this study is the structural complexity of biomedical knowledge. Unlike general-domain facts, which are often one-to-one (e.g., a city has one capital), medical knowledge is predominantly one-to-many. A single disease (subject) often has multiple valid treatments or symptoms (objects).

The researchers found that 90.4% of long-tail biomedical knowledge is one-to-many, compared to 82% for popular knowledge. This complexity significantly degrades the performance of editing algorithms.

Performance comparison between one-to-one and one-to-many biomedical knowledge in BioMedLM across different co-occurrence bins.

As shown in the data, the accuracy gap between one-to-one and one-to-many knowledge is staggering—reaching 42.19% for the rarest facts in an edited BioMedLM. Current editing methods are optimized for "pinning" a single association; when faced with the inherent ambiguity of multiple valid clinical paths, they struggle to generalize effectively.

Conclusion and Future Directions

While knowledge editing is a powerful tool for updating LLMs, it is not yet a "silver bullet" for rare clinical knowledge. The persistent performance gap and the challenge of one-to-many relationships highlight the need for domain-specific editing strategies.

Future work should focus on developing algorithms that can handle the inherent complexity of medical taxonomies. Furthermore, expanding the evaluation to include clinical notes and textbooks beyond PubMed abstracts will be crucial for ensuring these models are safe and reliable for real-world clinical decision support.

Summary updated

BibTeX
@misc{yi2025can,
  author = {Xinhao Yi and Jake Lever and Kevin Bryson and Zaiqiao Meng},
  title = {{Can We Edit LLMs for Long-Tail Biomedical Knowledge?}},
  year = {2025},
  eprint = {2504.10421},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2504.10421},
}
Figure