Introduction
Named Entity Recognition (NER) is a fundamental task in natural language processing. While standard NER assumes that entities are non-overlapping (flat), many real-world applications—particularly in the biomedical and news domains—encounter nested entities. For instance, in the phrase "[mouse [IL-2R alpha] protein] gene," the inner span "IL-2R alpha" is a Protein entity, while the entire span is a DNA entity.
Traditional approaches often struggle with this complexity. Sequence labeling methods typically require complex, error-prone transformations to flatten structures. Conversely, span-based models, which classify every possible sub-segment, suffer from high computational costs ($O(n^2)$) and often neglect explicit boundary information. This paper proposes a Supervised Multi-Head Self-Attention (SMHSA) network to address these challenges by treating NER as a multi-class classification of word pairs, assigning specific attention heads to individual entity types.
Methodology
The proposed framework utilizes a multi-task learning approach consisting of two primary modules: Boundary Detection (BD) and Entity Classification (EC).
Boundary Detection (BD) Module
The BD module identifies potential entity boundaries using a BiLSTM followed by a Feed-Forward Neural Network (FFNN). It employs a tagging scheme (BIEO: Beginning, Inside, Ending, Outside) to provide explicit signals regarding where entities start and stop. This module helps the overall network focus on valid spans and reduces the search space for the classification module.
Entity Classification (EC) Module
The core of the system is the EC module, which leverages the SMHSA layer. Unlike standard Transformer attention where heads learn arbitrary patterns, each head in SMHSA is supervised to correspond to exactly one entity type (e.g., DNA, RNA, Protein).
As illustrated in the architecture, the model calculates correlation intensities between "head" and "tail" word representations within type-specific subspaces. If the correlation for a specific head exceeds a predefined threshold $\delta$, the span is identified as belonging to that entity type. This "type-aware" mechanism allows the model to recognize that a word's role (e.g., as a start or end token) can change depending on the entity category being considered.
Datasets and Evaluation
The model was rigorously evaluated on five benchmark datasets to test its performance on both nested and flat NER:
- Nested NER: ACE2004, ACE2005 (General news/web), and GENIA (Biomedical).
- Flat NER: JNLPBA (Biomedical) and CoNLL03-English (News).
The inclusion of flat NER datasets demonstrates the architecture's generalizability beyond specialized nested structures.
Key Results
The SMHSA model, particularly when augmented with pre-trained BERT embeddings (MHSA-BERT), achieved state-of-the-art (SOTA) results across all tested benchmarks.
As shown in the results table, the MHSA-BERT model achieved F1 scores of 86.3% on ACE2004, 85.4% on ACE2005, and 79.6% on GENIA. These figures represent a significant improvement over previous benchmarks, such as Tan et al. (2020), by approximately 1.0% to 1.5%. Furthermore, the model reached an F1 score of 93.6% on the CoNLL03 flat NER dataset, proving its robustness in standard labeling scenarios.
Ablation studies confirmed that both the Boundary Detection module and the use of contextual embeddings (BERT) are critical to the system's success. Removing the BD module led to a noticeable decline in precision, highlighting the importance of explicit boundary supervision.
Discussion and Conclusion
The SMHSA network offers an efficient and effective solution for nested NER by supervising attention heads to learn type-specific semantic correlations. By treating the problem as pairwise word classification rather than exhaustive span enumeration, the model maintains computational efficiency while capturing complex overlapping structures.
Future work may involve exploring type embeddings to model the semantic relationships between different entity categories directly. Additionally, addressing the "long-tail" challenge in biomedical datasets like GENIA—where entities are often longer and more complex—remains a priority for further refining the architecture's performance in specialized domains.
Summary updated