CLaC at SemEval-2023 Task 2: Comparing Span-Prediction and Sequence-Labeling Approaches for NER

Harsh Verma, Sabine BerglerACL 2023

TL;DR

This paper compares two popular approaches for Named Entity Recognition (NER) – sequence labeling and span prediction – on the complex MultiCoNER 2 task. The authors found that their span prediction system slightly outperformed sequence labeling, and that using a larger language model (XLM-ROBERTa large) and special input tokens significantly boosted performance for both methods.

What Problem Does This Paper Address?

Named Entity Recognition (NER) is the task of identifying and classifying named entities (like people, organizations, locations) in text. The MultiCoNER 2 shared task specifically focuses on identifying complex, fine-grained named entities within short, potentially noisy sentences, which makes it a challenging benchmark. This task requires distinguishing between 30 different entity types and handling errors like typos. The paper aims to compare two dominant NER paradigms – sequence labeling (where each token is classified) and span prediction (where entire entity spans are identified and classified) – on this demanding task.

Methods

The researchers employed two main approaches, both leveraging the XLM-ROBERTa large language model:

  • Sequence Labeling: This standard approach treats NER as a token-level classification problem. Each token in the input sentence is assigned a label from the BIO (Beginning, Inside, Outside) scheme, extended with specific entity type suffixes (e.g., B-Corp, I-Corp). The model predicts a BIO tag for each token, and these predictions are then decoded into word-level entities.
  • Span Prediction: This method identifies and classifies entire spans of text as entities. The authors represent each potential span by concatenating the embeddings of its start and end tokens. A linear layer then predicts the entity type for each span, including a "Neg_Span" label for non-entities. A crucial part of this method involves a decoding step to resolve overlapping span predictions, prioritizing longer spans and randomly selecting between partially overlapping ones.

Both systems were fine-tuned using the Adam optimizer with a learning rate of 1e-5 and a batch size of 1, employing early stopping.

Datasets

The primary dataset used is MultiCoNER 2, a shared task dataset focused on fine-grained and noisy named entity recognition. The competition data was tokenized and provided with gold labels in the BIO format.

Key Insights and Ideas

  • Span Prediction Edges Out Sequence Labeling: For this specific task, the span prediction approach demonstrated a slight performance advantage over sequence labeling, suggesting its suitability for identifying complex and potentially out-of-vocabulary entities.
  • Larger Models Matter: Utilizing the larger XLM-ROBERTa large model provided a significant performance boost compared to its base version, highlighting the importance of model capacity for complex NER.
  • Special Tokens Enhance Performance: Incorporating special input tokens (like <s> and </s>) from XLM-ROBERTa into the input significantly improved the performance of both sequence labeling and span prediction models, particularly boosting the sequence labeling system.
  • Post-Competition Improvements: Increasing the batch size and adding special token embeddings after the initial competition submission led to notable performance gains, demonstrating the impact of hyperparameter tuning and architectural choices.

Key Results

The paper reports that their best span prediction system achieved a macro F1 score of 59.0 on the test set, slightly outperforming their best sequence labeling system (56.9). Post-competition experiments showed that increasing the batch size from 1 to 4 improved results by at least 3.2% for system B. Adding embeddings for special tokens (systems B+E) further improved performance, with the sequence model gaining 2.1% and nearly closing the gap with the span-based model. On the validation set, span prediction models initially showed higher performance than on the test set.

Limitations and Future Work

The authors acknowledge that their submission did not use any external knowledge. They also note that post-competition improvements, such as increasing batch size and adding special tokens, significantly impacted performance, suggesting that further tuning could yield even better results. The paper implicitly suggests that exploring more sophisticated methods for handling noisy data and complex entity structures could be fruitful future directions.

Summary updated

BibTeX
@inproceedings{verma2023clac,
  author = {Harsh Verma and Sabine Bergler},
  title = {{CLaC at SemEval-2023 Task 2: Comparing Span-Prediction and Sequence-Labeling Approaches for NER}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure