Hierarchically-Refined Label Attention Network for Sequence Labeling

Leyang Cui, Yue Zhang

Leyang Cui, Yue Zhang Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP-IJCNLP)

Abstract

The Hierarchically-Refined Label Attention Network (LAN) is a novel alternative to the industry-standard BiLSTM-CRF for sequence labeling tasks like POS tagging and NER. By replacing the computationally expensive and locally-constrained CRF layer with a multi-layer attention mechanism over label embeddings, LAN achieves higher accuracy and significantly faster inference speeds, particularly on tasks with large label sets.

Introduction

Sequence labeling—the process of assigning a tag to every word in a sentence—is a cornerstone of natural language processing. For years, the "gold standard" architecture has been the BiLSTM-CRF. In this setup, a BiLSTM encodes the text, and a Conditional Random Field (CRF) layer models the dependencies between tags. However, the CRF approach has two major flaws: it typically only models dependencies between immediate neighbors (the Markov constraint), and its decoding complexity is quadratic relative to the number of possible labels ($O(|L|^2n)$). This computational cost makes it slow for tasks with large label sets, such as CCG supertagging.

To address these limitations, the authors propose the Label Attention Network (LAN). This model captures global label dependencies more effectively than a CRF while maintaining linear computational complexity ($O(|L|n)$).

Methodology

The core innovation of LAN is treating labels as learnable embeddings in a continuous space. This allows the model to learn that certain tags are semantically similar, helping it generalize better than a CRF, which treats every tag pair transition as a unique parameter.

Model Architecture

Instead of a single output layer, the model uses stacked "attentive BiLSTM" layers. At each layer, the model uses the BiLSTM hidden states as a "query" to perform multi-head attention over the label embeddings. This produces a probability distribution over labels for each word at every stage of the hierarchy.

The architecture of the Label Attention Network (LAN), showing the hierarchical stacking of BiLSTM and Label Attention Inference layers.

Hierarchical Refinement

Each layer in the LAN produces a marginal label distribution. Higher layers "refine" these distributions by looking at the distributions produced by the layer below across the entire sentence. Because the attention mechanism can observe the label distributions of the entire sentence from the previous layer, it can capture long-distance constraints that a standard linear-chain CRF would miss.

An illustration of the hierarchical refinement process. The model corrects the tag for "can" from a Modal (MD) to a Verb (VB) as it moves from the first to the last layer by leveraging global context.

Experimental Results

The model was evaluated across four diverse sequence labeling tasks: English POS Tagging (WSJ), Multilingual POS Tagging (UD v2.2), Named Entity Recognition (OntoNotes 5.0), and CCG Supertagging.

  • Accuracy: LAN outperformed BiLSTM-CRF across the board. On WSJ POS tagging, it reached 97.65% accuracy. On CCG supertagging, it achieved 94.7%, matching or beating state-of-the-art systems.
  • Efficiency: For CCG supertagging, which involves a large label set, LAN was nearly 20% faster at decoding than the CRF baseline due to its linear complexity.
  • Interpretability: Using t-SNE to visualize the label embeddings, the authors found that the model naturally clusters similar parts of speech together during training (e.g., all verb forms move toward the same area of the vector space).

Conclusion and Limitations

While LAN offers significant improvements in accuracy and efficiency, the authors noted that it converges more slowly during training than a CRF, requiring more iterations to reach peak performance. Additionally, performance is sensitive to the size of the label embeddings. Despite these trade-offs, the hierarchical attention framework represents a scalable and powerful alternative for fine-grained structured prediction problems in NLP.

Summary updated

BibTeX
@inproceedings{cui2019hierarchically,
  author = {Leyang Cui and Yue Zhang},
  title = {{Hierarchically-Refined Label Attention Network for Sequence Labeling}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2019},
  url = {https://openreview.net/forum?id=Li4ePqIG01_},
}
Figure