How Do Your Biomedical Named Entity Recognition Models Generalize to Novel Entities?

Hyunjae Kim, Jaewoo Kang

Introduction

The rapid expansion of biomedical research, with over 3,000 papers published daily, necessitates robust Biomedical Named Entity Recognition (BioNER) systems. These models are essential for identifying mentions of drugs, diseases, and genes to facilitate automated knowledge extraction. However, current state-of-the-art models often suffer from a hidden flaw: they excel at memorizing entities seen during training but struggle to generalize to novel synonyms or entirely new concepts.

The Generalization Gap

Standard BioNER benchmarks often provide a misleading sense of security. High F1 scores frequently mask a model's reliance on "statistical shortcuts"—memorizing specific words like "diabetes" rather than understanding the linguistic context. This paper identifies an "overestimation trap," noting that over 60% of entities in common test sets are already present in the training data.

A stark example of this failure is seen with BioBERT. Despite achieving over 90% recall on standard benchmarks, it recognized only 3.4% of COVID-19 mentions when trained on pre-2020 datasets (like BC5CDR). Because "COVID-19" did not follow the morphological patterns of "typical" disease names (e.g., ending in "-itis" or "syndrome"), the model failed to identify it as a disease.

Methodology: Partitioning and Debiasing

To systematically address this, the authors propose partitioning test data into three categories based on Concept Unique Identifiers (CUIs):

  1. Memorization (Mem): Mentions and concepts seen during training.
  2. Synonym Generalization (Syn): New surface forms (names) for concepts seen during training (e.g., recognizing "Advil" after seeing "Ibuprofen").
  3. Concept Generalization (Con): Entirely new concepts and names never encountered during training.

To bridge the performance gap, a Statistics-based Debiasing Method (Bias Product) was introduced. This approach identifies words highly correlated with specific labels in the training set. By calculating these prior distributions and debiasing the model during training, the neural network is forced to prioritize linguistic context over simple word-label associations.

Experimental Results

The study utilized the NCBI-disease and BC5CDR benchmarks, alongside a custom COVID-19 dataset consisting of 5,000 PubMed sentences. Key findings include:

  • Performance Drop: On the BC5CDR dataset, BioBERT’s recall dropped from 93.3% on memorized mentions to approximately 74% for synonyms and new concepts.
  • Debiasing Impact: The Bias Product method consistently improved performance on "Syn" and "Con" splits. On the NCBI-disease "Con" split, BioBERT’s recall improved from 84.1% to 86.1%.
  • Rare Entity Recognition: The debiased model significantly improved the detection of rare diseases with unique naming structures, such as "African Iron Overload" or "Pinta," sometimes by more than 10 percentage points.

Discussion and Limitations

While debiasing enhances a model's ability to discover novel entities, it introduces a trade-off. The authors noted a tendency toward "over-exploration," where the model occasionally misidentifies non-noun phrases (e.g., "Loss of") or adjectives as entities because it is no longer strictly bound by training set statistics.

Future research should focus on refining these debiasing techniques to maintain high precision while exploring novel entities, potentially by incorporating syntactic constraints or leveraging larger-scale unlabeled biomedical corpora.

Summary updated

Abstract

The number of biomedical literature on new biomedical concepts is rapidly increasing, which necessitates a reliable biomedical named entity recognition (BioNER) model for identifying new and unseen entity mentions. However, it is questionable whether existing models can effectively handle them. In this work, we systematically analyze the three types of recognition abilities of BioNER models: memorization, synonym generalization, and concept generalization. We find that although current best models achieve state-of-the-art performance on benchmarks based on overall performance, they have limitations in identifying synonyms and new biomedical concepts, indicating they are overestimated in terms of their generalization abilities. We also investigate failure cases of models and identify several difficulties in recognizing unseen mentions in biomedical literature as follows: (1) models tend to exploit dataset biases, which hinders the models’ abilities to generalize, and (2) several biomedical names have novel morphological patterns with weak name regularity, and models fail to recognize them. We apply a statistics-based debiasing method to our problem as a simple remedy and show the improvement in generalization to unseen mentions. We hope that our analyses and findings would be able to facilitate further research into the generalization capabilities of NER models in a domain where their reliability is of utmost importance.
BibTeX
@article{kim2022how,
  author = {Hyunjae Kim and Jaewoo Kang},
  title = {{How Do Your Biomedical Named Entity Recognition Models Generalize to Novel Entities?}},
  journal = {IEEE Access},
  year = {2022},
  doi = {10.1109/access.2022.3157854},
}
Figure