A Supervised Multi-Head Self-Attention Network for Nested Named Entity Recognition

Yongxiu Xu, Heyan Huang, Chong Feng, Yue Hu

Introduction

Named Entity Recognition (NER) is a fundamental task in natural language processing. While standard NER assumes that entities are non-overlapping (flat), many real-world applications—particularly in the biomedical and news domains—encounter nested entities. For instance, in the phrase "[mouse [IL-2R alpha] protein] gene," the inner span "IL-2R alpha" is a Protein entity, while the entire span is a DNA entity.

Traditional approaches often struggle with this complexity. Sequence labeling methods typically require complex, error-prone transformations to flatten structures. Conversely, span-based models, which classify every possible sub-segment, suffer from high computational costs ($O(n^2)$) and often neglect explicit boundary information. This paper proposes a Supervised Multi-Head Self-Attention (SMHSA) network to address these challenges by treating NER as a multi-class classification of word pairs, assigning specific attention heads to individual entity types.

Methodology

The proposed framework utilizes a multi-task learning approach consisting of two primary modules: Boundary Detection (BD) and Entity Classification (EC).

Architecture of the proposed SMHSA network showing Boundary Detection and Entity Classification modules.

Boundary Detection (BD) Module

The BD module identifies potential entity boundaries using a BiLSTM followed by a Feed-Forward Neural Network (FFNN). It employs a tagging scheme (BIEO: Beginning, Inside, Ending, Outside) to provide explicit signals regarding where entities start and stop. This module helps the overall network focus on valid spans and reduces the search space for the classification module.

Entity Classification (EC) Module

The core of the system is the EC module, which leverages the SMHSA layer. Unlike standard Transformer attention where heads learn arbitrary patterns, each head in SMHSA is supervised to correspond to exactly one entity type (e.g., DNA, RNA, Protein).

As illustrated in the architecture, the model calculates correlation intensities between "head" and "tail" word representations within type-specific subspaces. If the correlation for a specific head exceeds a predefined threshold $\delta$, the span is identified as belonging to that entity type. This "type-aware" mechanism allows the model to recognize that a word's role (e.g., as a start or end token) can change depending on the entity category being considered.

Datasets and Evaluation

The model was rigorously evaluated on five benchmark datasets to test its performance on both nested and flat NER:

  • Nested NER: ACE2004, ACE2005 (General news/web), and GENIA (Biomedical).
  • Flat NER: JNLPBA (Biomedical) and CoNLL03-English (News).

The inclusion of flat NER datasets demonstrates the architecture's generalizability beyond specialized nested structures.

Key Results

The SMHSA model, particularly when augmented with pre-trained BERT embeddings (MHSA-BERT), achieved state-of-the-art (SOTA) results across all tested benchmarks.

Performance comparison on nested NER datasets (ACE2004, ACE2005, GENIA).

As shown in the results table, the MHSA-BERT model achieved F1 scores of 86.3% on ACE2004, 85.4% on ACE2005, and 79.6% on GENIA. These figures represent a significant improvement over previous benchmarks, such as Tan et al. (2020), by approximately 1.0% to 1.5%. Furthermore, the model reached an F1 score of 93.6% on the CoNLL03 flat NER dataset, proving its robustness in standard labeling scenarios.

Ablation studies confirmed that both the Boundary Detection module and the use of contextual embeddings (BERT) are critical to the system's success. Removing the BD module led to a noticeable decline in precision, highlighting the importance of explicit boundary supervision.

Discussion and Conclusion

The SMHSA network offers an efficient and effective solution for nested NER by supervising attention heads to learn type-specific semantic correlations. By treating the problem as pairwise word classification rather than exhaustive span enumeration, the model maintains computational efficiency while capturing complex overlapping structures.

Future work may involve exploring type embeddings to model the semantic relationships between different entity categories directly. Additionally, addressing the "long-tail" challenge in biomedical datasets like GENIA—where entities are often longer and more complex—remains a priority for further refining the architecture's performance in specialized domains.

Summary updated

Abstract

In recent years, researchers have shown an increased interest in recognizing the overlapping entities that have nested structures. However, most existing models ignore the semantic correlation between words under different entity types. Considering words in sentence play different roles under different entity types, we argue that the correlation intensities of pairwise words in sentence for each entity type should be considered. In this paper, we treat named entity recognition as a multi-class classification of word pairs and design a simple neural model to handle this issue. Our model applies a supervised multi-head self-attention mechanism, where each head corresponds to one entity type, to construct the word-level correlations for each type. Our model can flexibly predict the span type by the correlation intensities of its head and tail under the corresponding type. In addition, we fuse entity boundary detection and entity classification by a multitask learning framework, which can capture the dependencies between these two tasks. To verify the performance of our model, we conduct extensive experiments on both nested and flat datasets. The experimental results show that our model can outperform the previous state-of-the-art methods on multiple tasks without any extra NLP tools or human annotations.
BibTeX
@inproceedings{xu2021supervised,
  author = {Yongxiu Xu and Heyan Huang and Chong Feng and Yue Hu},
  title = {{A Supervised Multi-Head Self-Attention Network for Nested Named Entity Recognition}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.1609/aaai.v35i16.17669},
}
Figure