Improving Tagging Consistency and Entity Coverage for Chemical Identification in Full-text Articles

Hyunjae Kim, Mujeen Sung, Wonjin Yoon, Sungjoon Park, Jaewoo KangarXiv 2021 · OpenReview

Introduction

While most biomedical Named Entity Recognition (NER) research focuses on titles and abstracts, the core of scientific discovery often resides within the full text of research articles. This transition from short summaries to long-form text introduces two primary challenges: tagging inconsistency and a significant increase in "unseen" entities. Researchers from Korea University and UC San Diego addressed these issues in their winning entry for the BioCreative VII Track 2 challenge, developing a system that prioritizes document-level consistency and broad entity coverage.

Methodology

The team's pipeline consists of an NER module followed by a normalization stage. To tackle the problem of inconsistent tagging—where a chemical might be identified in one paragraph but missed in another—the researchers employed a straightforward majority voting post-processing strategy. This method reconciles predictions across the entire document, ensuring that if a term is identified as a chemical in a significant portion of the text, it is consistently labeled throughout.

To address the "unseen" entity problem (where nearly half of the chemicals in full-text test sets may not appear in training data), the system utilizes transfer learning and model ensembling.

The system architecture leverages transfer learning from multiple datasets and an ensemble of models to improve recognition of unseen chemicals.

As shown in the architecture, models were pre-trained on established datasets like BC5CDR and CHEMDNER, as well as a synthetic dataset, NLM-Chem-Syn, created by replacing entities with synonyms. For the final normalization step, a hybrid approach was used, combining a high-precision dictionary lookup with a neural model (BioSyn with SapBERT) to capture complex linguistic variations.

Results and Key Insights

The proposed system secured 1st place in the BioCreative VII Track 2 NER task, outperforming over 80 other submissions. The team's top-performing run achieved a Precision of 0.8759, a Recall of 0.8587, and an F1 score of 0.8672.

Final rankings for the BioCreative VII Track 2 challenge, highlighting the team's first-place performance across multiple runs.

Key findings from the study include:

  • The Power of Simplicity: The majority voting step, despite its simplicity compared to complex global attention mechanisms, improved the F1 score by up to 1.0 point and successfully corrected massive inconsistencies in long documents.
  • Full-Text Vocabulary Gap: The choice of base Language Model (LM) is critical. Models trained on full-text PubMed articles (Bio-LM-large) significantly outperformed those trained only on abstracts (BioBERT), highlighting the distinct linguistic style of full-text literature.
  • Synthetic Data Impact: Using synonym replacement to generate synthetic training data proved to be a highly effective, low-cost method for improving the model's ability to generalize to novel chemical entities.

Conclusion

The success of this system at BioCreative VII underscores the importance of addressing document-level context and vocabulary diversity in full-text NER. While challenges remain in minimizing error propagation during the normalization phase, the combination of majority voting, transfer learning, and domain-specific language models provides a robust framework for extracting chemical information from the vast body of scientific literature.

Summary updated

Abstract

This paper is a technical report on our system submitted to the chemical identification task of the BioCreative VII Track 2 challenge. The main feature of this challenge is that the data consists of full-text articles, while current datasets usually consist of only titles and abstracts. To effectively address the problem, we aim to improve tagging consistency and entity coverage using various methods such as majority voting within the same articles for named entity recognition (NER) and a hybrid approach that combines a dictionary and a neural model for normalization. In the experiments on the NLM-Chem dataset, we show that our methods improve models' performance, particularly in terms of recall. Finally, in the official evaluation of the challenge, our system was ranked 1st in NER by significantly outperforming the baseline model and more than 80 submissions from 16 teams.
BibTeX
@misc{kim2021improving,
  author = {Hyunjae Kim and Mujeen Sung and Wonjin Yoon and Sungjoon Park and Jaewoo Kang},
  title = {{Improving Tagging Consistency and Entity Coverage for Chemical Identification in Full-text Articles}},
  year = {2021},
  eprint = {2111.10584},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2111.10584},
}
Figure