IITD at SemEval-2023 Task 2: A Multi-Stage Information Retrieval Approach for Fine-Grained Named Entity Recognition

Shivani Choudhary, Niladri Chatterjee, Subir SahaACL 2023

TL;DR

This paper tackles the challenging task of fine-grained Named Entity Recognition (NER) across multiple languages, especially for ambiguous and complex entities. They propose a novel multi-stage information retrieval (IR) pipeline that augments language models with relevant contextual passages, leading to improved NER performance, particularly on challenging entity types.

What Problem Does This Paper Address?

Named Entity Recognition (NER) is a fundamental task in Natural Language Processing (NLP) that involves identifying and classifying named entities in text (e.g., people, organizations, locations). While traditional NER systems perform well on common entity types, they struggle with:

  • Fine-grained classification: Distinguishing between numerous, highly specific entity categories (e.g., differentiating between various types of corporations or creative works).
  • Ambiguous and complex entities: Entities that can have multiple meanings or are difficult to define precisely.
  • Low-context scenarios: Identifying entities when the surrounding text is limited, making disambiguation harder.
  • Multilingualism: Handling these challenges across diverse languages.

The MultiCoNER-II competition specifically addresses these issues by introducing a fine-grained NER task with 36 different classes and a dataset that includes noisy examples to simulate real-world conditions. The paper aims to improve NER performance by effectively leveraging external information through an IR approach.

Methods

The core of the proposed approach is a multi-stage information retrieval (IR) pipeline designed to enhance the performance of language models for fine-grained NER. Here's a breakdown of their method:

  1. Information Retrieval (IR) Pipeline:
    • Document Indexing: They build an inverted index of documents from two large corpora: MS-MARCO and KILT.
  2. Initial Retrieval: A BM25-based model is used to retrieve relevant documents for a given query (sentence).
  3. Passage Chunking and Re-ranking: Retrieved documents are segmented into smaller passages. A T5-based pre-trained re-ranking model is then employed to identify the most relevant passages, which are then used as additional context.
  4. Named Entity Recognition (NER) Stack:
    • Encoder: A transformer-based encoder (specifically BERT-large-uncased and BERT-base-uncased) generates contextual embeddings for the input sentence and the retrieved passages.
  5. Linear Layer: This layer projects the embeddings to a dimension corresponding to the number of NER classes (73, likely including BIO tags).
  6. CRF Layer: A Conditional Random Field (CRF) layer is used to learn transition probabilities between tags, improving sequence labeling.
  7. Training Strategy:
    • Ensemble of Losses: To handle class imbalance and improve robustness, they use a weighted combination of Focal Loss and CRF Loss. Focal loss is particularly useful for down-weighting easy examples and focusing on hard ones, which is beneficial for low-performing classes.
  8. Warmup Schedule: A warmup schedule followed by a constant learning rate is used during training.

Datasets

  • MultiCoNER V2: This is the primary dataset used for training and evaluation, featuring 36 fine-grained NER classes and including noisy data to simulate real-world challenges.
  • MS-MARCO: A large dataset of search queries and passages, used for building the document index for the IR pipeline.
  • KILT: A benchmark for knowledge-intensive NLP tasks, also used for building the document index.

Key Insights and Ideas

  • Leveraging External Context is Crucial: The paper strongly demonstrates that augmenting language models with relevant passages retrieved through an IR pipeline significantly improves NER performance, especially for challenging entity types.
  • Focal Loss for Imbalance: The introduction of Focal Loss is key to addressing the inherent class imbalance in fine-grained NER datasets, leading to better performance on under-represented entity classes.
  • Multi-Stage IR for Relevance: A multi-stage IR approach, including passage re-ranking, is more effective than simple document retrieval for extracting pertinent contextual information.
  • Noisy Data Impact: The performance drop on the noisy test set highlights the real-world challenge of handling imperfect data and the need for robust models.

Key Results

  • The proposed system ranked 15th in the English track of the MultiCoNER-II competition.
  • A performance gain of 0.0877 F1 score was observed when using the retrieved context compared to a baseline BERT-large + CRF model.
  • The system achieved a macro@F1 score of 0.6752 on the clean test set and 0.5359 on the noisy test set.
  • The paper also shows that using a variable weight scheme for Focal Loss (specifically, softmax over class weights) yielded a slightly better F1 score (0.718) compared to constant or inverse weighting.

Limitations and Future Work

  • Performance on Noisy Data: The model's performance significantly degrades on the noisy test set, indicating a need for more robust handling of corrupted data.
  • Computational Constraints: The authors were unable to evaluate with a full 512 context length due to computing constraints, suggesting that larger context windows might yield further improvements.
  • Future Work Suggestions:
    • Explore query reformulation techniques to better handle short contexts and noise.
  • Investigate further the use of Focal Loss with masking of context tokens.
  • Extend the IR pipeline to handle more complex query reformulation scenarios.

Summary updated

BibTeX
@inproceedings{choudhary2023iitd,
  author = {Shivani Choudhary and Niladri Chatterjee and Subir Saha},
  title = {{IITD at SemEval-2023 Task 2: A Multi-Stage Information Retrieval Approach for Fine-Grained Named Entity Recognition}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure