Why Attention? Analyze BiLSTM Deficiency and Its Remedies in the Case of NER

Peng-Hsuan Li, Tsu-Jui Fu, Wei-Yun MaAAAI 2020 · OpenReview

Abstract

BiLSTMs have long served as the backbone for Named Entity Recognition (NER) systems. However, this paper identifies a fundamental "XOR limitation" in standard BiLSTM architectures that prevents them from effectively modeling interactions between past and future contexts. We demonstrate that simple "cross-structures"—specifically self-attention and interleaved BiLSTM layers—can remedy this deficiency. Our evaluation on OntoNotes 5.0 and WNUT 2017 benchmarks reveals significant performance gains, particularly for complex, multi-token entities, with improvements of up to 8.7% on long mentions.

The XOR Limitation of BiLSTMs

In standard sequence labeling, a BiLSTM processes text in two independent directions: a forward LSTM encodes past context, and a backward LSTM encodes future context. At each time step, these representations are typically concatenated or added.

We mathematically prove that this linear combination creates an XOR limitation. Because the forward and backward states are processed independently, the model cannot capture patterns that require simultaneous knowledge of both specific past and specific future contexts. If a tagging decision relies on a non-linear interaction between the left and right context (similar to an XOR logic gate), a standard BiLSTM is theoretically incapable of resolving it. Notably, adding a Conditional Random Field (CRF) layer does not solve this, as CRF transitions are independent of the word sequence and cannot fix the underlying encoder's representational failure.

Proposed Remedies: Cross-Structures

To overcome this bottleneck, we propose architectures that allow past and future contexts to interact at each time step through cross-structures:

  1. Cross-BiLSTM-CNN (Additive): This model interleaves forward and backward LSTM layers. Higher-level layers take inputs from both directions of the lower layers, allowing the network to build a holistic representation of the sequence in a feed-forward manner.
  2. Att-BiLSTM-CNN (Multiplicative): This model adds a multi-head self-attention mechanism on top of a standard BiLSTM. By using dot-product attention, the model correlates past and future contexts multiplicatively, enabling tokens to dynamically query their surroundings.

Analysis: How Attention Solves the Boundary Problem

Visualizing self-attention weights reveals that different attention heads naturally specialize to solve the entity boundary problem. Some heads focus on immediate preceding tokens to signal the end of an entity, while others look for succeeding tokens to identify the start.

Visualization of different attention heads specializing in identifying entity boundaries and internal dependencies across different layers.

This specialization allows the model to move beyond the "single-token fallback" seen in standard BiLSTMs. When faced with ambiguous surface forms, standard models often default to predicting single-token entities. Cross-structures provide the contextual confidence needed to accurately bound and classify multi-token mentions.

Key Results

Our models outperformed strong BiLSTM-CNN baselines by 0.7% F1 on OntoNotes and 2.2% on the noisy WNUT 2017 dataset. The impact is most pronounced on difficult entity types and long mentions.

Comparison of entity recognition between the baseline BiLSTM and the Attention-BiLSTM model, showing improved classification of multi-token entities.
  • Multi-Token Mentions: Accuracy on entities of length 3 or more improved by up to 8.7%.
  • Noisy Text Robustness: On the WNUT dataset, which features emerging entities and user-generated text, the cross-structure models showed superior generalization compared to memorization-heavy baselines.

Conclusion

While state-of-the-art NER often relies on massive external resources like BERT or ELMo, our work focuses on fixing the core sequence encoder. By addressing the XOR limitation through simple architectural changes, we provide a more robust foundation for sequence labeling. Integrating these cross-structures into resource-heavy SOTA models represents a promising path for future NER improvements.

Summary updated

Abstract

BiLSTM has been prevalently used as a core module for NER in a sequence-labeling setup. State-of-the-art approaches use BiLSTM with additional resources such as gazetteers, language-modeling, or multi-task supervision to further improve NER. This paper instead takes a step back and focuses on analyzing problems of BiLSTM itself and how exactly self-attention can bring improvements. We formally show the limitation of (CRF-)BiLSTM in modeling cross-context patterns for each word – the XOR limitation. Then, we show that two types of simple cross-structures – self-attention and Cross-BiLSTM – can effectively remedy the problem. We test the practical impacts of the deficiency on real-world NER datasets, OntoNotes 5.0 and WNUT 2017, with clear and consistent improvements over the baseline, up to 8.7% on some of the multi-token entity mentions. We give in-depth analyses of the improvements across several aspects of NER, especially the identification of multi-token mentions. This study should lay a sound foundation for future improvements on sequence-labeling NER1.
BibTeX
@inproceedings{li2020why,
  author = {Peng-Hsuan Li and Tsu-Jui Fu and Wei-Yun Ma},
  title = {{Why Attention? Analyze BiLSTM Deficiency and Its Remedies in the Case of NER}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  url = {https://openreview.net/forum?id=mtsrzibFGl},
}
Figure