RGAT at SemEval-2023 Task 2: Named Entity Recognition Using Graph Attention Network

Abir ChakrabortyACL 2023

TL;DR

This paper introduces RGAT, a novel Graph Attention Network (GAT) approach for multilingual Named Entity Recognition (NER) on complex, short, and low-contextual texts. By incorporating dependency parsing information into a GAT, RGAT aims to better capture the nuanced relationships between words for more accurate entity identification, achieving a respectable 23rd rank in the English track of SemEval-2023 Task 2.

What Problem Does This Paper Address?

Named Entity Recognition (NER) is a fundamental NLP task, but existing benchmarks often use well-formed sentences with limited entity classes. Real-world applications, however, frequently involve short, noisy texts with many granular entities and little contextual information, making traditional NER models struggle. The SemEval-2023 Task 2 (MultiCoNER II) specifically targets this challenge by providing a dataset with 12 languages, 66 fine-grained entity classes, and simulated errors, requiring models to be robust to ambiguity and limited context.

Methods

The core of the RGAT approach is its use of a Relational Graph Attention Network (RGAT). Unlike standard NER models that rely solely on token embeddings and sequential information, RGAT leverages the dependency parse tree of the input sentence. Each word is treated as a node in a graph, and the dependency relations between words are represented as edges. The RGAT then iteratively updates node embeddings by attending to both neighboring nodes and the type of dependency relation connecting them. This graph-based representation is combined with standard features like BERT embeddings and part-of-speech tags. The authors also experimented with adding BiLSTM and Transformer layers before a final CRF layer to further enhance the model's capabilities.

Datasets

The primary dataset used in this work is the SemEval-2023 Task 2: MultiCoNER II dataset. This dataset covers 12 languages and features 66 fine-grained entity classes across 6 broad groups. The authors also evaluated their model on two standard NER datasets: CoNLL-2003 and WNUT-2017 for comparative analysis.

Key Insights and Ideas

  • Dependency Structure is Key for Complex NER: The paper strongly suggests that incorporating syntactic dependency information is crucial for handling the complexity and ambiguity present in short, low-contextual texts, especially when dealing with a large number of fine-grained entity classes.
  • RGAT Captures Relational Nuances: The RGAT architecture's ability to explicitly model relational information between words, beyond just sequential proximity, appears to be a significant advantage for this challenging NER task.
  • External Knowledge is a Major Boost: The authors acknowledge that approaches using external knowledge bases (like Wikipedia) significantly outperformed theirs, highlighting a clear avenue for future improvement.
  • Sentence Length Matters: Performance generally improves with increasing sentence length, suggesting that even a small amount of context can be beneficial for the model.

Key Results

On the English track of the SemEval-2023 Task 2 (MultiCoNER II), the RGAT-BERT-BiLSTM-CRF model achieved a fine-grained macro-averaged F1-score of 56.91%, securing 23rd rank. The clean-subset F1 was 61.29%. When evaluated on standard NER datasets, the RGAT model showed competitive performance, particularly on the CoNLL-2003 dataset.

Limitations and Future Work

The authors explicitly state that their approach heavily relies on the availability of dependency structures, which can be a limitation for languages where robust dependency parsers are not readily available or for poorly formed sentences. They also acknowledge that their model's performance was significantly lower than those that incorporated external knowledge bases. Future work will focus on exploring optimal ways to integrate external data sources to further enhance the model's accuracy.

Summary updated

BibTeX
@inproceedings{chakraborty2023rgat,
  author = {Abir Chakraborty},
  title = {{RGAT at SemEval-2023 Task 2: Named Entity Recognition Using Graph Attention Network}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure