L3i at SemEval-2022 Task 11: Straightforward Additional Context for Multilingual Named Entity Recognition

Emanuela Boros, Carlos-Emiliano González-Gallardo, Jose Moreno, Antoine DoucetNAACL 2022

TL;DR

This paper tackles the challenge of Named Entity Recognition (NER) in complex, multilingual, and low-resource scenarios, particularly for short texts. The authors propose a straightforward method of augmenting input sentences with semantically relevant contexts from the training data, which shows promising improvements, especially when using uncased language models and combining language-specific and multilingual models.

What Problem Does This Paper Address?

Named Entity Recognition (NER) is a fundamental task in Natural Language Processing (NLP) that involves identifying and classifying named entities (like people, organizations, and locations) within text. While NER has seen significant advancements, especially with Transformer-based models like BERT, several challenges remain. These include:

  • Short Texts: NER on snippets like web queries or social media posts is difficult due to limited context.
  • Complex Entities: Identifying entities that are long noun phrases or have specific formatting (e.g., dates in historical documents) is challenging.
  • Multilingual and Code-Mixed Data: Handling texts that contain entities in different languages than the main text, or texts that mix multiple languages, adds significant complexity.
  • Low-Resource Languages: Many NER systems are developed for high-resource languages like English, leaving a gap for languages with less available data.

The SemEval-2022 Task 11, "Multilingual Complex Named Entity Recognition (MultiCoNER)," specifically addresses these challenges by focusing on 11 languages, including low-resource ones, and incorporating short, low-context, and potentially code-mixed data.

Methods

The core of the proposed approach is to enhance the context available to the NER model. The authors employ a two-pronged strategy:

  1. Language-Specific and Multilingual Models: For each language, they utilize a language-specific BERT-based pre-trained model. To further improve performance, especially for multilingual and code-mixed scenarios, they also incorporate a general multilingual language model (like XLM-RoBERTa-large). This combination aims to leverage both specialized and broad linguistic knowledge.
  2. Context Augmentation: The key novelty lies in augmenting each input sentence with relevant contexts retrieved from the training set. This is done using a multilingual Sentence-BERT (SBERT) model to generate sentence embeddings and then finding the top-10 most semantically similar sentences from the training data. Two variations of context augmentation are explored:
  3. "context w/ entities": The retrieved contexts are directly concatenated with the original sentence.
  4. "context w/o entities": The entities within the retrieved contexts are masked with a special token (<ENT>) to avoid confusing the model by providing explicit entity labels in the context. This aims to improve contextual understanding without directly revealing entity information.

The final NER model is built upon a BERT encoder with a Conditional Random Field (CRF) layer on top, a standard architecture for sequence labeling tasks.

Datasets

The paper uses the SemEval-2022 Task 11 (MultiCoNER) dataset. This dataset comprises:

  • Monolingual Datasets: For 11 languages (English, Spanish, Dutch, Russian, Turkish, Korean, Farsi, German, Chinese, Hindi, and Bangla), each with fixed-sized training and development sets and large test sets.
  • Multilingual Dataset: A concatenation of all monolingual sets.
  • Code-Mixed Dataset: A smaller dataset specifically designed for code-mixed scenarios.

The paper notes that the test sets are significantly larger than the training and development sets, which is a deliberate design choice by the organizers to test out-of-domain performance.

Key Insights and Ideas

  • Context is King (Especially Unmasked): The "context w/o entities" approach generally yielded better results than "context w/ entities." This suggests that providing semantically relevant context without explicitly revealing entities can be more beneficial for the model to learn entity boundaries and types.
  • Multilingual Models Boost Performance: Combining language-specific models with a general multilingual model consistently improved performance across most languages. This highlights the value of leveraging cross-lingual knowledge, especially for low-resource languages.
  • Uncased Models Matter: The authors prioritized using uncased language models, which proved effective. This is a practical consideration for NER systems dealing with potentially noisy or inconsistent casing in short texts.
  • Topic Overlap and Performance Correlation: A surprising finding was the negative correlation between topic similarity between train/dev/test sets and NER performance. Higher topic similarity (indicating more overlap in domains) often led to lower F1 scores, suggesting that the model might struggle when the training and testing domains are too similar, or that the "out-of-domain" nature of test sets is crucial for evaluation.

Key Results

The paper reports that their proposed methods outperform the baseline scores provided by the organizers across most languages.

  • High-Resource Languages: The approach achieved above-average results for English, Spanish, and German.
  • Low-Resource and Multilingual: Performance was lower than average for some low-resource languages (Hindi, Bangla) and the multilingual/mixed sets. This is partly attributed to not having specialized language models for Hindi and Bangla, relying instead on the multilingual model.
  • Contextual Improvements: Adding context, particularly "context w/o entities," led to considerable improvements for a larger number of languages compared to just using the language-specific and multilingual models.

Limitations and Future Work

  • Low-Resource Language Gap: The authors acknowledge that performance for Hindi and Bangla was lower, partly due to the lack of specialized language models.
  • Topic Similarity Paradox: The observed negative correlation between topic similarity and performance is an interesting area for further investigation.
  • Future Work: The authors suggest future work could focus on the automatic generation of semantically relevant contexts without the presence of entities, further refining the "context w/o entities" approach. They also plan to explore the influence of test set size on performance.

Summary updated

BibTeX
@inproceedings{boros2022l,
  author = {Emanuela Boros and Carlos-Emiliano González-Gallardo and Jose Moreno and Antoine Doucet},
  title = {{L3i at SemEval-2022 Task 11: Straightforward Additional Context for Multilingual Named Entity Recognition}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure