Attention-based Multi-level Feature Fusion for Named Entity Recognition

Zhiwei Yang, Hechang Chen, Jiawei Zhang, Jing Ma, Yi Chang

Abstract

The AMFF (Attention-based Multi-level Feature Fusion) framework improves Named Entity Recognition (NER) by simultaneously capturing local and global information at both the character and word levels. By using attention mechanisms to weight these diverse perspectives, the model excels at disambiguating polysemous words—like "Washington"—that can represent different entity types depending on the broader context.

Problem Statement: The Challenge of Polysemy

Named Entity Recognition is a cornerstone of Natural Language Processing, yet it frequently stumbles on "polysemy"—words that have multiple meanings or categories. For instance, "Washington" could be a person (George Washington), a location (Washington State), or part of an organization (Washington University).

Figure 1: Illustration of the polysemy problem where "Washington" can represent different entity types based on local and global context.

Traditional models often rely on local context windows or simple character-level embeddings. While effective, these methods often miss "global" cues hidden elsewhere in a sentence or fail to integrate specific syntactic information like capitalization and lexical phrases. The authors argue that a truly robust NER system needs to look at the text through multiple lenses simultaneously: from the fine-grained shapes of characters to the broad relationships between distant words.

Methodology: The AMFF Framework

The authors introduce the AMFF framework, which processes input text through four parallel "feature selection" components:

  1. Global Character-level: Uses a BiLSTM combined with self-attention to capture long-distance dependencies between characters across the entire sequence.
  2. Local Character-level: Employs a CNN with max-pooling to extract morphological features, such as capitalization, which are strong indicators for entities.
  3. Global Word-level: Utilizes self-attention on word embeddings to identify lexical phrases and keywords (e.g., recognizing that the word "in" often precedes a location).
  4. Local Word-level: Distills the most prominent features from the word-level attention output using max-pooling and a feed-forward network.
Figure 2: The architecture of the AMFF framework, showing the parallel feature selection branches and the fusion layer.

Once these four feature sets are extracted, the model uses an automatic fusion strategy. It assigns a learnable weight ($\lambda$) to each component, allowing the model to decide which "lens" is most important for a given dataset. Finally, these fused features are fed into a standard BiLSTM-CRF (Conditional Random Field) layer to predict the final sequence of entity tags.

Experimental Setup and Datasets

The model was evaluated on four diverse benchmark datasets to test its generalizability across different domains:

  • CoNLL-2003: A standard benchmark for general-purpose NER (News).
  • NCBI-disease: A medical dataset focused on identifying disease names.
  • SciERC: A dataset derived from scientific AI paper abstracts.
  • JNLPBA: A technical bio-entity dataset.

Key Insights and Results

The AMFF model consistently outperformed state-of-the-art baselines across all four datasets. It achieved an F1 score of 94.48% on CoNLL-2003 and 92.11% on NCBI-disease.

A key "aha moment" of this research is demonstrating that global attention can solve local ambiguity. In the sentence "Washington University... is located in Missouri," the word "University" might be far from "Washington," but the global attention mechanism links them, correctly identifying "Washington" as part of an Organization (ORG) rather than a Person (PER). Through ablation studies, the researchers also found that word-level features generally contributed more to the final score than character-level features, likely because pre-trained word embeddings already carry significant semantic weight.

Conclusion and Future Work

While the AMFF framework is powerful, the authors note that the architecture is designed to be extensible. Future work could involve adding specific linguistic features like affixes or part-of-speech tags. Additionally, integrating external knowledge bases (like Wikipedia or domain-specific ontologies) could further improve performance on highly technical or rare entities. While the parallel feature extraction is effective, the trade-off between this added complexity and inference speed remains an area for further exploration.

Summary updated

Abstract

Named entity recognition (NER) is a fundamental task in the natural language processing (NLP) area. Recently, representation learning methods (e.g., character embedding and word embedding) have achieved promising recognition results. However, existing models only consider partial features derived from words or characters while failing to integrate semantic and syntactic information (e.g., capitalization, inter-word relations, keywords, lexical phrases, etc.) from multi-level perspectives. Intuitively, multi-level features can be helpful when recognizing named entities from complex sentences. In this study, we propose a novel framework called attention-based multi-level feature fusion (AMFF), which is used to capture the multi-level features from different perspectives to improve NER. Our model consists of four components to respectively capture the local character-level, global character-level, local word-level, and global word-level features, which are then fed into a BiLSTM-CRF network for the final sequence labeling. Extensive experimental results on four benchmark datasets show that our proposed model outperforms a set of state-of-the-art baselines.
BibTeX
@inproceedings{yang2020attention,
  author = {Zhiwei Yang and Hechang Chen and Jiawei Zhang and Jing Ma and Yi Chang},
  title = {{Attention-based Multi-level Feature Fusion for Named Entity Recognition}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.24963/ijcai.2020/497},
}
Figure