Abstract
Biomedical Entity Linking (BioEL) is a critical task for extracting structured knowledge from unstructured clinical and research text. However, the field suffers from fragmented evaluation practices, making it difficult to compare model performance objectively. This paper introduces a unified benchmarking framework to evaluate nine state-of-the-art BioEL models across seven diverse datasets. Our findings reveal that while contextualized models like ArboEL lead in accuracy, significant gaps remain in handling gene/protein entities and generalizing to unseen data.
Introduction
In biomedical research, ambiguity is pervasive. An acronym like "AD" can refer to Alzheimer’s Disease, Atopic Dermatitis, or Actinomycin D depending on the context. For automated systems, resolving these mentions to unique identifiers in knowledge bases like the Unified Medical Language System (UMLS) is known as Entity Linking (EL). Despite numerous claims of state-of-the-art (SOTA) performance, the lack of standardized datasets and preprocessing steps has hindered progress. Kartchner et al. address this by providing a comprehensive "reality check" through a unified evaluation framework.
The Challenges of Biomedical Entity Linking
BioEL is uniquely difficult due to several factors:
- Synonymy and Polysemy: Entities often have numerous aliases, and single terms frequently refer to multiple distinct concepts.
- Data Scarcity: High-quality labeled data requires expensive domain expertise.
- Evaluation Fragmentation: Inconsistent use of datasets and metrics makes "apples-to-apples" comparisons nearly impossible for practitioners.
Methodology: A Unified Framework
The researchers categorized nine SOTA models into three architectural families:
- Alias Matching: (e.g., SapBERT, SciSpacy) focusing on string similarity and name embeddings.
- Contextualized EL: (e.g., ArboEL, ClusterEL) utilizing bi-encoders to represent both mentions and entity descriptions within their context.
- Autoregressive EL: (e.g., BioBART) generative models that produce the canonical name of an entity.
The evaluation follows a two-stage pipeline: Candidate Generation (CG) to narrow the search space, followed by Named Entity Disambiguation (NED) to select the final entity.
Experimental Results
The study utilized seven benchmark datasets, including MedMentions, BC5CDR, and GNormPlus, unified via the BigBio framework.
Performance Leaders
ArboEL emerged as the accuracy champion, particularly on the MedMentions dataset. However, simpler models like SapBERT showed surprising robustness and significantly higher inference speeds, making them attractive for production environments.
The "Gene Gap" and Abbreviation Resolution
A major finding was the "Gene Gap": all models performed poorly on gene and protein mentions compared to chemicals or diseases. This is largely due to cross-species naming conventions that current models fail to disambiguate. Conversely, implementing a simple abbreviation resolution step led to performance gains of up to 69.5% for abbreviated entities, highlighting a straightforward path for system improvement.
Discussion and Future Work
While current models excel in specific domains, zero-shot generalization to unseen entities remains a significant weakness. Furthermore, the researchers noted that many SOTA models suffer from poor reproducibility due to incomplete documentation. While Large Language Models (LLMs) offer potential for zero-shot tasks, their tendency to hallucinate and high computational costs remain barriers. The authors have open-sourced their framework to serve as a standardized proving ground for future BioEL research.
Summary updated