Abstract
Knowledge editing has emerged as an effective approach for updating large language models (LLMs) by modifying their internal knowledge. However, their application to the biomedical domain faces unique challenges due to the long-tailed distribution of biomedical knowledge, where rare and infrequent information is prevalent. In this paper, we conduct the first comprehensive study to investigate the effectiveness of knowledge editing methods for editing long-tail biomedical knowledge. Our results indicate that, while existing editing methods can enhance LLMs' performance on long-tail biomedical knowledge, their performance on long-tail knowledge remains inferior to that on high-frequency popular knowledge, even after editing. Our further analysis reveals that long-tail biomedical knowledge contains a significant amount of one-to-many knowledge, where one subject and relation link to multiple objects. This high prevalence of one-to-many knowledge limits the effectiveness of knowledge editing in improving LLMs' understanding of long-tail biomedical knowledge, highlighting the need for tailored strategies to bridge this performance gap.
The Long-Tail Challenge in Biomedicine
In the biomedical field, knowledge follows a stark "long-tail" distribution. While popular topics like Type 1 Diabetes are documented in thousands of papers, rare conditions like Evans Syndrome may only appear in a handful of documents. Large Language Models (LLMs) often fail to capture this infrequent information during pre-training, leading to hallucinations or silence on critical niche topics. This study investigates whether Knowledge Editing (KE)—techniques designed to update specific facts without full retraining—can bridge the performance gap between popular and rare medical knowledge.
The CliKT Benchmark and Methodology
To evaluate these methods, the researchers introduced CliKT (Clinical Knowledge Triples), a benchmark extracted from SNOMED CT. Using an entity-linking pipeline across 37 million PubMed abstracts, they categorized over 100,000 clinical triples by their "co-occurrence number." Facts appearing in fewer than 10 documents were classified as "long-tail."
The study evaluated several state-of-the-art editing methods:
- Locate-and-Edit (ROME & MEMIT): Modifies specific "knowledge memory" layers (typically MLPs).
- Meta-Learning (MEND): Uses a separate network to learn fast, localized weight updates.
- In-Context Editing (IKE): Provides new knowledge via prompt demonstrations.
- Fine-Tuning (FT): Standard gradient-descent on a single layer.
Key Insights: The One-to-Many Hurdle
A primary discovery of this research is the prevalence of "one-to-many" relationships in long-tail data. Approximately 90.4% of long-tail biomedical knowledge consists of relations where one subject (e.g., a disease) links to multiple valid objects (e.g., various treatments). LLMs struggle with these because they are often biased toward a single, most frequent answer encountered during training.
As shown in the analysis, there is a persistent performance gap: for the rarest facts, accuracy for one-to-one knowledge was initially 42.19% higher than for one-to-many. Even after applying the most effective editing methods, a significant gap remains, suggesting that current KE techniques are not yet optimized for complex medical relationships.
Results and Performance Gap
- Pre-edit Failure: Before editing, BioMedLM’s accuracy on long-tail facts was 22.86% lower than its accuracy on popular facts.
- ROME as the Leader: ROME proved most effective, improving BioMedLM’s long-tail accuracy by approximately 52%.
- Generalization Decline: While methods like ROME are excellent at memorizing specific facts (high Reliability), their ability to generalize that knowledge to rephrased questions drops significantly as the fact becomes rarer.
- Open-Source Edge: Updated open-source models (like BioMedLM with ROME) actually outperformed closed-source models like GPT-4o-mini on these specific long-tail tasks.
Conclusion
While knowledge editing offers a path toward more reliable clinical LLMs, the "structural" advantage of high-frequency knowledge persists. Future work must address the one-to-many problem, perhaps through Retrieval-Augmented Generation (RAG) or structure-aware finetuning objectives that explicitly teach models to handle multiple valid outcomes for a single medical query.
Summary updated