Comparing and combining some popular NER approaches on Biomedical tasks

Harsh Verma, Sabine Bergler, Narjesossadat TahaeiACL 2023

TL;DR

This paper explores and combines three popular Named Entity Recognition (NER) approaches (SEQ, SeqCRF, and SpanPred) on biomedical datasets. They demonstrate that combining these models, particularly through a novel "Meta" system, can significantly improve performance, especially in recall, while maintaining high precision and F1 scores.

What Problem Does This Paper Address?

Named Entity Recognition (NER) is a crucial task in natural language processing, aiming to identify and classify named entities (like diseases, genes, or drugs) within text. While traditional NER approaches often frame this as a sequence labeling problem (assigning a label to each token), more recent work has explored a "span-prediction" paradigm. The challenge lies in understanding how these different approaches perform on specialized domains like biomedicine and how their strengths and weaknesses can be leveraged through combination. This paper aims to compare these popular NER methods on biomedical tasks and investigate effective ways to combine them to achieve superior performance.

Methods

The paper compares three distinct NER approaches:

  • SEQ (Sequence Labeling): A straightforward model that uses a language model to get token representations and then a linear layer to classify each token.
  • SeqCRF (Sequence Labeling with Conditional Random Fields): Similar to SEQ, but it adds a Conditional Random Field (CRF) layer on top of the token representations. This layer helps model dependencies between adjacent labels, often improving sequence labeling performance.
  • SpanPred (Span Prediction): This approach treats NER as identifying and classifying spans of text. It represents each potential span by concatenating the embeddings of its start and end tokens and then uses a linear layer to classify the span.

The authors also explore two simple combination methods:

  • Union: A set union of predictions from individual models.
  • MajVote (Majority Voting): Predictions that are agreed upon by a majority of the individual models.

Finally, they introduce a novel Meta combiner system. This system learns to identify and filter out incorrect predictions from individual models, aiming to boost precision without sacrificing recall. Meta uses special "markers" to highlight potential entities and prompts to indicate the entity type, learning from predictions on a validation set.

Datasets

The paper evaluates the models on four biomedical NER datasets:

  • GENIA: A well-established dataset for biomedical text.
  • NCBI-Disease: Focuses on disease names.
  • LivingNER (Spanish): A Spanish dataset for biomedical entities.
  • SocialDisNER (Spanish): Another Spanish dataset, specifically for social media text related to diseases.

Key Insights and Ideas

  • SpanPred's Strength: The SpanPred model shows state-of-the-art performance on the LivingNER and SocialDisNER datasets, highlighting its effectiveness for certain biomedical tasks.
  • SeqCRF's Competitiveness: The SeqCRF model also achieves state-of-the-art results on the same Spanish datasets, demonstrating the continued relevance of CRF-based sequence labeling.
  • Combination is Key: Combining models, even with simple methods like majority voting, can yield significant improvements.
  • Meta's Promise: The novel Meta combiner system shows particular promise in improving recall and maintaining high F1 scores, effectively learning to correct individual model errors.
  • Domain Specificity: The performance of models can vary across different biomedical datasets, suggesting the need for careful model selection and adaptation.

Key Results

  • SpanPred improved F1 by 1.3 and 0.6 on LivingNER and SocialDisNER, respectively.
  • SeqCRF improved F1 by 0.2 and 0.7 on LivingNER and SocialDisNER, respectively.
  • The Meta combiner system significantly boosted F1 (+1.2) and recall (+2.1) over the combined systems on the GENIA dataset.
  • MajVote consistently achieved high precision and F1 across all four datasets.
  • Union consistently boosted recall but often at the cost of precision.

Limitations and Future Work

  • Meta's Precision Focus: The Meta combiner is designed to improve precision by filtering incorrect predictions. While it can improve recall, its primary strength lies in reducing false positives.
  • Computational Cost: While the paper focuses on simple and popular approaches, the combination methods, especially Meta, introduce additional computational overhead.
  • Future Work: The authors suggest further exploration of Meta's capabilities, particularly in its ability to combat the precision drop seen with the Union method. They also mention the potential for more sophisticated combination strategies. The release of their code is a significant contribution for future research.

Summary updated

BibTeX
@inproceedings{verma2023comparing,
  author = {Harsh Verma and Sabine Bergler and Narjesossadat Tahaei},
  title = {{Comparing and combining some popular NER approaches on Biomedical tasks}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure