RIGA at SemEval-2023 Task 2: NER Enhanced with GPT-3

Eduards Mukans, Guntis BarzdinsACL 2023

TL;DR

This paper describes the RIGA team's approach to the SemEval-2023 Task 2: Multilingual Complex Named Entity Recognition (MultiCoNER II) challenge. They achieved a significant performance boost by leveraging GPT-3 for "in-context learning" to gather additional contextual information for sentences, which was then used to fine-tune a pre-trained neural network. This method led to a 17% improvement in results and a 12th place ranking out of 34 teams.

What Problem Does This Paper Address?

The MultiCoNER II task presents a significant challenge in Named Entity Recognition (NER) by requiring the classification of words into 36 complex named and nominal entities. Unlike traditional NER, which often deals with broad categories like "person" or "location," MultiCoNER demands finer-grained distinctions (e.g., "artist," "athlete," "politician"). Furthermore, the task's datasets are characterized by lower-cased text and inherent "noisiness," including typographical errors, missing labels, and inaccurate labels, making accurate entity recognition considerably more difficult. Traditional NER systems often struggle with these complexities, especially when relying solely on gazetteers or knowledge bases, which can be prone to errors and miss entities outside their scope.

Methods

The RIGA team's approach centers on augmenting existing pre-trained language models (LLMs) with additional context derived from GPT-3. The core methodology involves three main steps:

  1. Data Preprocessing: The original dataset was cleaned and converted into a format suitable for HuggingFace Datasets.
  2. Gathering Context with GPT-3: Instead of building custom knowledge bases, the team utilized GPT-3's "in-context learning" capability. This involved crafting prompt templates to guide GPT-3 in generating relevant contextual information for each sentence. They employed the text-davinci-003 model and carefully tuned parameters like "temperature" (set to 0.2 for predictability) and "max length" (set to 96) to ensure precise and robust context generation. The "presence penalty" was set to 0 to encourage token repetition, which can aid in entity identification.
  3. NER Model Training: The team experimented with various training strategies. Initially, they used adapter modules on top of a bert-base-uncased model. They also explored training without early stopping, a technique that proved beneficial in previous work. Ultimately, they scaled up their model to xlm-roberta-large and performed full fine-tuning without early stopping, which yielded the best results.

Datasets

The primary dataset used was the MultiCoNER II Task dataset. This dataset is notable for its "noisy" nature, containing typographical errors, missing labels, and inaccurate labels across 12 languages. For their submission, the RIGA team focused exclusively on the English track. They also created a "Cleaned" validation dataset by removing duplicate sentences from the original training data to mitigate potential bias.

Key Insights and Ideas

  • GPT-3 as a Contextual Augmenter: The paper strongly demonstrates that leveraging powerful LLMs like GPT-3 for "in-context learning" can significantly enhance performance on complex NER tasks, especially in noisy environments. This approach bypasses the need for extensive custom knowledge base development.
  • The Value of "Noisy" Data: While noisy data presents a challenge, the paper suggests that LLMs are adept at handling it. The significant performance boost observed when using GPT-3-generated contexts highlights the model's ability to infer correct information even with imperfect input.
  • Beyond Early Stopping: The findings reinforce the idea that traditional early stopping techniques might prematurely halt training, potentially hindering model performance. Continuing training, even past the point of apparent overfitting, can lead to more robust and accurate predictions, as seen in their "full-tuned-no-early" model.
  • Scaling Laws in Practice: The paper validates the effectiveness of "scaling laws" in improving NER performance. By scaling model size (xlm-roberta-large) and compute time (full fine-tuning without early stopping), they achieved substantial gains.

Key Results

The RIGA team's submission achieved a significant 17% boost in performance by incorporating GPT-3-generated contexts. Their final submission ranked 12th out of 34 teams in the overall English track. Notably, their performance on the noisy subset was particularly strong, ranking 7th. The best-performing model, full-tuned-no-early (using xlm-roberta-large and full fine-tuning without early stopping), achieved an F1 score of 79.7% on the validation dataset.

Limitations and Future Work

  • LLaMA Comparison: The authors briefly mention evaluating alternative language models like LLaMA after the competition. They found that LLaMA's generated contexts exhibited lower semantic coherence compared to GPT-3's outputs, suggesting GPT-3's superiority for this specific task.
  • Computational Resources: While the paper highlights the benefits of scaling, it implicitly acknowledges the significant computational resources required for such endeavors, especially when using large models like GPT-3 and performing full fine-tuning.
  • Future Work: The paper suggests that leveraging LLMs for context augmentation is a promising direction for improving NER performance, especially in challenging, noisy, and fine-grained scenarios. The authors also emphasize the continued importance of understanding and applying "scaling laws" for system optimization.

Summary updated

BibTeX
@inproceedings{mukans2023riga,
  author = {Eduards Mukans and Guntis Barzdins},
  title = {{RIGA at SemEval-2023 Task 2: NER Enhanced with GPT-3}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure