Generative Biomedical Entity Linking via Knowledge Base-Guided Pre-training and Synonyms-Aware Fine-tuning

Hongyi Yuan, Zheng Yuan, Sheng Yu

Abstract

Entities lie in the heart of biomedical natural language understanding, and the biomedical entity linking (EL) task remains challenging due to the fine-grained and diversiform concept names. Generative methods achieve remarkable performances in general domain EL with less memory usage while requiring expensive pre-training. Previous biomedical EL methods leverage synonyms from knowledge bases (KB) which is not trivial to inject into a generative method. In this work, we use a generative approach to model biomedical EL and propose to inject synonyms knowledge in it. We propose KB-guided pre-training by constructing synthetic samples with synonyms and definitions from KB and require the model to recover concept names. We also propose synonyms-aware fine-tuning to select concept names for training, and propose decoder prompt and multi-synonyms constrained prefix tree for inference. Our method achieves state-of-the-art results on several biomedical EL tasks without candidate selection which displays the effectiveness of proposed pre-training and fine-tuning strategies.

Introduction

Entity Linking (EL) is a critical component of biomedical natural language processing, involving the mapping of textual mentions (e.g., "Reiter syndrome") to unique identifiers in a Knowledge Base (KB), such as the Unified Medical Language System (UMLS). This task is particularly challenging in the medical domain due to high linguistic variability; a single clinical concept often possesses numerous synonyms, ranging from formal Latinate terms to colloquial abbreviations.

Historically, EL has been dominated by discriminative models that utilize dense vector spaces to match mentions to concepts. However, these models suffer from high memory overhead when scaling to millions of entities. Generative models, such as BART-based architectures, offer a more memory-efficient alternative by treating EL as a sequence-to-sequence task. Despite their potential, generative models typically require vast amounts of human-labeled data and often struggle with the "lexical gap"—the discrepancy between a mention in the text and the specific canonical name the model is expected to generate. This paper introduces GenBioEL, a framework designed to bridge these gaps through synthetic knowledge injection and similarity-aware training.

Methodology

The GenBioEL framework utilizes a BART-large architecture and introduces three primary innovations to enhance generative entity linking:

1. KB-Guided Pre-training

To address the scarcity of labeled biomedical data, the authors developed a synthetic pre-training strategy. By leveraging the UMLS knowledge base, the model is trained on templates that associate synonyms with their definitions. For instance, a training sample might require the model to complete a sequence like: [BOS] [ST] Reiter syndrome [ET] is defined as [Definition of C0085435] [EOS]. This approach allows the model to learn the semantic relationships between diverse names for the same entity without requiring manually annotated corpora.

2. Synonyms-Aware Fine-tuning

Standard generative models often struggle when forced to generate a specific canonical name that differs significantly from the input mention. GenBioEL mitigates this by selecting the training target based on textual similarity. Using character 3-gram TF-IDF similarity, the model identifies the synonym within the KB that most closely matches the mention in the source text. By reducing the distance between input and output, the optimization landscape becomes smoother, allowing the model to learn more effectively.

3. Constrained Generative Inference

To ensure that the model generates valid entities, GenBioEL employs a multi-synonym constrained prefix tree during the decoding phase. This ensures the model only outputs strings that exist as valid synonyms within the target KB. Once a valid name is generated, it is mapped back to its corresponding Concept Unique Identifier (CUI).

Experimental Evaluation

Datasets

The model was pre-trained on a subset of UMLS containing 2.37 million concepts and evaluated across four diverse benchmarks:

  • BC5CDR: Chemical and disease mentions from PubMed.
  • NCBI-disease: Disease name recognition and normalization.
  • COMETA: Layperson medical terms sourced from social media (Reddit).
  • AskAPatient (AAP): Non-standard medical language from patient forums.

Results and Key Findings

GenBioEL achieved state-of-the-art (SOTA) results for Recall@1 across several datasets, demonstrating the power of generative models in specialized domains:

  • BC5CDR: 93.3% (a 1.4% improvement over previous SOTA).
  • COMETA: 81.4% (a 1.3% improvement).
  • Zero-Shot Performance: The model showed exceptional generalization on unseen concepts, outperforming the best baseline by nearly 10% (86.9% vs. 77.5%).

Furthermore, GenBioEL proved to be highly efficient. It achieved these results with significantly fewer resources than general-domain generative models, requiring only 6 GPU days for pre-training compared to the 32 GPU days typically required by models like GENRE.

Discussion and Conclusion

The success of GenBioEL underscores two major insights: first, that rich Knowledge Bases can serve as a viable proxy for human-annotated data through synthetic pre-training; and second, that aligning training targets with input similarity (the "path of least resistance") is crucial for generative tasks.

While the model excels at zero-shot generalization and memory efficiency, challenges remain regarding the handling of exceptionally long mentions and the model's inherent dependency on the completeness of the KB's synonym list. Future work may explore the integration of a retrieval-based candidate selection step to further refine the model's accuracy in complex scenarios.

Summary updated

Abstract

Entities lie in the heart of biomedical natural language understanding, and the biomedical entity linking (EL) task remains challenging due to the fine-grained and diversiform concept names. Generative methods achieve remarkable performances in general domain EL with less memory usage while requiring expensive pre-training. Previous biomedical EL methods leverage synonyms from knowledge bases (KB) which is not trivial to inject into a generative method. In this work, we use a generative approach to model biomedical EL and propose to inject synonyms knowledge in it. We propose KB-guided pre-training by constructing synthetic samples with synonyms and definitions from KB and require the model to recover concept names. We also propose synonyms-aware fine-tuning to select concept names for training, and propose decoder prompt and multi-synonyms constrained prefix tree for inference. Our method achieves state-of-the-art results on several biomedical EL tasks without candidate selection which displays the effectiveness of proposed pre-training and fine-tuning strategies.
BibTeX
@inproceedings{yuan2022generative,
  author = {Hongyi Yuan and Zheng Yuan and Sheng Yu},
  title = {{Generative Biomedical Entity Linking via Knowledge Base-Guided Pre-training and Synonyms-Aware Fine-tuning}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
  url = {https://openreview.net/forum?id=WkdPResXXlI},
}
Figure