Hierarchically-Refined Label Attention Network for Sequence Labeling

Leyang Cui, Yue Zhang

Abstract

Conditional Random Fields (CRFs) have long been the standard for statistical sequence labeling. However, in neural sequence labeling, the BiLSTM-CRF architecture does not always outperform simpler BiLSTM-softmax models. This is likely due to the limited information gain provided by the simple Markov label transition model of CRFs over strong neural encoders. We investigate a hierarchically-refined label attention network (LAN) that explicitly leverages label embeddings and captures long-term dependencies through incrementally refined label distributions. Our results across POS tagging, NER, and CCG supertagging demonstrate that LAN improves accuracy while significantly increasing processing speed compared to BiLSTM-CRF.

Introduction

Sequence labeling—the task of assigning a tag to every word in a sentence—is foundational to NLP tasks like Part-of-Speech (POS) tagging and Named Entity Recognition (NER). For years, the BiLSTM-CRF model has been the dominant approach, using a CRF layer to ensure global consistency in label predictions. However, CRFs are constrained by Markov assumptions, typically only considering immediate neighbors, and suffer from high computational complexity ($O(|L|^2n)$) when dealing with large label sets.

The Label Attention Network (LAN)

The proposed BiLSTM-LAN architecture replaces the rigid CRF layer with a series of hierarchically stacked Label Attention Network layers. Each layer consists of a BiLSTM encoding sublayer and a label-attention inference sublayer.

The architecture of the BiLSTM-LAN, showing the hierarchical stacking of encoding and label-attention inference layers.

Unlike traditional models, LAN treats labels as trainable embedding vectors. It employs a multi-head attention mechanism where word features act as "queries" and label embeddings act as "keys" and "values." This allows the model to learn semantic relationships between tags and capture dependencies across the entire sentence, effectively breaking the Markov constraint.

Hierarchical Refinement

A core innovation of the LAN is its hierarchical structure. The model does not produce a final prediction in a single pass. Instead, each successive layer takes the label distributions from the previous layer as input, allowing for incremental refinement of the predictions.

An example of hierarchical refinement: label distributions for 'They can fish...' are updated from the first layer to the last layer.

As illustrated above, lower layers may be ambiguous about specific tags (e.g., whether "fish" is a noun or a verb). Higher layers use the refined context from surrounding words to resolve these ambiguities, leading to more accurate global predictions.

Experimental Results

The LAN was evaluated on several benchmarks, including the Penn Treebank (WSJ) for POS tagging, OntoNotes 5.0 for NER, and CCGBank for CCG supertagging.

  • Accuracy: LAN achieved a state-of-the-art 97.65% accuracy on WSJ POS tagging and an 88.16 F1-score on NER, outperforming BiLSTM-CRF baselines.
  • Efficiency: By avoiding the $O(|L|^2)$ complexity of the CRF transition matrix, LAN is significantly faster, especially on tasks with large label sets. In CCG supertagging (425 tags), LAN processed 713 sentences per second compared to 599 for the CRF.
  • Convergence: While LAN eventually reaches higher accuracy, it typically requires more training epochs than a CRF to converge, as it must learn complex label embeddings and attention weights.

Conclusion

The Hierarchically-Refined Label Attention Network represents a significant step forward for sequence labeling. By replacing the Markov chain with hierarchical attention over label embeddings, it captures long-range dependencies more effectively than a CRF while offering superior computational efficiency. This approach is particularly beneficial for complex tasks with large label vocabularies.

Summary updated

Abstract

CRF has been used as a powerful model for statistical sequence labeling. For neural sequence labeling, however, BiLSTM-CRF does not always lead to better results compared with BiLSTM-softmax local classification. This can be because the simple Markov label transition model of CRF does not give much information gain over strong neural encoding. For better representing label sequences, we investigate a hierarchically-refined label attention network, which explicitly leverages label embeddings and captures potential long-term label dependency by giving each word incrementally refined label distributions with hierarchical attention. Results on POS tagging, NER and CCG supertagging show that the proposed model not only improves the overall tagging accuracy with similar number of parameters, but also significantly speeds up the training and testing compared to BiLSTM-CRF.
BibTeX
@inproceedings{cui2019hierarchically,
  author = {Leyang Cui and Yue Zhang},
  title = {{Hierarchically-Refined Label Attention Network for Sequence Labeling}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2019},
  url = {https://openreview.net/forum?id=Li4ePqIG01_},
}
Figure