Bi-directional context-aware network for the nested named entity recognition

Yao Li, Ningsheng Liao, He Yan, Zhang Ye, Xu WangScientific 2024 · DOI

TL;DR

This paper introduces Bi-DCAN, a novel network for Nested Named Entity Recognition (Nested NER) that addresses limitations of existing span-based models by explicitly modeling inter-span semantic relationships and incorporating relative positional information. Bi-DCAN achieves competitive results with fewer parameters, demonstrating improved accuracy in identifying complex, nested entities.

What Problem Does This Paper Address?

Named Entity Recognition (NER) is a fundamental task in Natural Language Processing (NLP) that involves identifying and classifying named entities (like people, organizations, locations) in text. A significant challenge in NER is "nested named entity recognition" (Nested NER), where entities can be contained within other entities, leading to overlapping or hierarchical structures. Traditional span-based NER models, while effective at capturing entity spans, often treat each span independently. This means they miss crucial semantic connections between different spans and overlook the precise positional information of an entity's start and end tokens. This oversight can lead to inaccuracies, especially with longer or more complex nested entities. The paper aims to bridge this gap by developing a model that can effectively capture these inter-span semantic relationships and positional nuances.

Methods

The core of the proposed solution is the Bi-Directional Context-Aware Network (Bi-DCAN). The authors introduce several key innovations:

  • Span-Level Semantic Relation Model: They propose a new way to represent semantic relationships between spans, moving beyond simple adjacency. This model considers both horizontal and vertical relationships in a "planarized" sentence representation, capturing how spans relate to each other in a 2D grid.
  • Bi-Directional Context-Aware Network (Bi-DCAN): This network is designed to learn these span-level semantic relationships. It employs a multi-granularity cross-attention mechanism that can simultaneously capture both local and long-distance semantic dependencies between spans. This is achieved through specialized "local attention" and "dilation attention" mechanisms.
  • Rotary Position Embedding (RoPE) with Bi-affine Mechanism: To address the issue of positional information, the authors integrate RoPE into the bi-affine mechanism. The bi-affine mechanism is a standard technique for generating span representations by considering pairwise interactions. By incorporating RoPE, the model can more accurately capture the relative positional information between the head (start) and tail (end) tokens of an entity. This is crucial for precisely defining entity boundaries, especially for longer entities.

Datasets

The model was evaluated on three publicly available Nested NER datasets:

  • ACE2004
  • ACE2005
  • GENIA

The paper does not introduce new datasets but uses these standard benchmarks to compare its performance against existing methods.

Key Insights and Ideas

  • Inter-span relationships are critical for Nested NER: The paper strongly argues that simply identifying individual spans is insufficient. Understanding how spans relate to each other semantically (e.g., containment, adjacency) is key to accurately resolving nested structures.
  • Positional information is more than just start/end indices: The relative positioning of head and tail tokens within a span, and how this relates to other spans, is a vital signal that traditional models often underutilize. RoPE helps unlock this signal.
  • A balanced attention approach is effective: The multi-granularity cross-attention mechanism, combining local and dilation attention, provides a sweet spot between capturing fine-grained local interactions and broader contextual dependencies without excessive computational cost.
  • Model efficiency matters: The proposed Bi-DCAN achieves strong performance with significantly fewer parameters than some state-of-the-art models, suggesting a more efficient architecture.

Key Results

  • Improved F1 Scores: The Bi-DCAN model achieved improved F1 scores compared to the latest model Diffusion-NER, with an increase of 0.24 on ACE2005 and 0.09 on GENIA.
  • Reduced Parameters: The model reduces parameters by 20 million compared to Diffusion-NER, indicating greater computational efficiency.
  • Competitive Performance: On the ACE2004 dataset, the model showed only a marginal deviation (0.11) from the optimal Diffusion-NER.
  • Effective for Long Entities: Experiments showed that the RoPE module significantly improves the accurate representation of head and tail tokens for long entities.
  • Strong Performance Across Nested Types: The model performed well across various nested entity types, including "flat," "nested," "inner," "outer," and "nesting" categories.

Limitations and Future Work

  • Limitations: The paper acknowledges that while their model performs well, there's always room for improvement. The analysis of semantic dependency strength suggests that performance can be influenced by the distance between spans, with optimal performance observed at specific dilation rates and local lengths (around 3).
  • Future Work: The authors express a desire to further research semantic relations between spans and explore more effective ways to integrate local dependencies and boundary information to enhance model performance even further. They also mention the scalability of their method, showing good performance across different pre-trained models (BERT-Small, BERT-base, BERT-large, RoBERTa-base).

Summary updated

Abstract

The Span-based model can effectively capture the complex entity structure in the text, thus becoming the mainstream model for nested named entity recognition (Nested NER) tasks. However, traditional Span-based models decode each entity span independently. They do not consider the semantic connections between spans or the entities' positional information, which limits their performance. To address these issues, we propose a Bi-Directional Context-Aware Network (Bi-DCAN) for the Nested NER. Specifically, we first design a new span-level semantic relation model. Then, the Bi-DCAN is implemented to capture this semantic relationship. Furthermore, we incorporate Rotary Position Embedding into the bi-affine mechanism to capture the relative positional information between the head and tail tokens, enabling the model to more accurately determine the position of each entity. Experimental results show that compared to the latest model Diffusion-NER, our model reduces 20M parameters and increases the F1 scores by 0.24 and 0.09 on the ACE2005 and GENIA datasets respectively, which proves that our model has an excellent ability to recognise nested entities.
BibTeX
@inproceedings{li2024bi,
  author = {Yao Li and Ningsheng Liao and He Yan and Zhang Ye and Xu Wang},
  title = {{Bi-directional context-aware network for the nested named entity recognition}},
  booktitle = {Scientific},
  year = {2024},
  doi = {10.1038/s41598-024-67114-6},
}
Figure