Structured Refinement for Sequential Labeling

Yiran Wang, Hiroyuki Shindo, Yuji Matsumoto, Taro WatanabeACL 2021 · OpenReview

TL;DR

This paper introduces a novel "structured refinement" mechanism for sequential labeling tasks like Named Entity Recognition (NER) and Part-of-Speech (POS) tagging. By employing a globally normalized attention mechanism that considers the entire label sequence, their model effectively filters out irrelevant information and captures long-term dependencies, leading to improved performance. They also propose practical tricks for faster CRF computation and better Chinese character embedding initialization.

What Problem Does This Paper Address?

Sequential labeling tasks are fundamental in Natural Language Processing (NLP), aiming to assign a label to each token in a sequence (e.g., identifying "New York" as a location). While previous methods have focused on extracting rich features or using hierarchical attention to refine representations, they often suffer from limited capacity to model long-term dependencies. Specifically, locally normalized attention mechanisms, which only consider the current time step, can hinder performance. The paper addresses this by proposing a method that leverages structured attention, specifically using the marginal probability from a Conditional Random Field (CRF), to better capture global label dependencies and improve representation refinement.

Methods

The core of the proposed method is a structured refinement mechanism that combines representation refinement with globally normalized structured attention.

  • Globally Normalized Structured Attention: Instead of using locally normalized attention (like softmax over current labels), the authors propose using the marginal probability derived from a CRF. This marginal probability, calculated using the forward-backward algorithm (or efficiently via automatic differentiation), considers all possible label sequences, thus capturing long-term dependencies more effectively. This is a key departure from prior work like Cui and Zhang (2019).
  • Representation Refinement: The model iteratively refines hidden representations. This is achieved by using a bidirectional LSTM encoder and a refinement layer that incorporates the structured attention. They also employ residual connections and layer normalization for better training stability, which they show to be superior to simple concatenation.
  • CRF Integration: While Cui and Zhang (2019) omitted CRFs for speed, this paper reintroduces them by leveraging the marginal probability. They propose two implementation tricks to accelerate CRF computation:
  • Parallel GPU computation: Using the associative property of matrix multiplication to compute AB and CD in parallel.
  • Pre-compiling argument indices: To handle variable sentence lengths efficiently without excessive padding.
  • Character Embeddings Initialization: For Chinese datasets, they propose a trick to initialize character embeddings by projecting pre-trained word embeddings using an orthogonal matrix. This leverages the fact that many Chinese characters are also single-character words.

Datasets

The paper evaluates their model on several standard datasets for NER and POS tagging:

  • English NER:
    • CoNLL 2003
    • OntoNotes 5.0
  • Chinese NER:
    • OntoNotes 5.0
    • OntoNotes 4.0
  • English POS Tagging:
    • Wall Street Journal (WSJ)
    • Universal Dependencies (UD) v2.2

Key Insights and Ideas

  • Global vs. Local Attention: The paper strongly argues that globally normalized attention (derived from CRF marginals) is crucial for capturing long-term label dependencies, outperforming locally normalized attention. This is a significant insight into the limitations of previous hierarchical attention models.
  • CRF Re-emergence: By cleverly accelerating CRF computation, the paper demonstrates that CRFs can still be beneficial for sequential labeling, especially when integrated with advanced neural architectures.
  • Weight Tying Benefits: Sharing label embeddings and the CRF transition matrix across layers significantly improves performance, suggesting that explicit supervision on target labels is more effective than relying solely on unobserved latent variables.
  • Refinement Layers: The authors find that one refinement layer is sufficient for their model, contrasting with previous work that required two, suggesting their structured attention is more potent.
  • Chinese Character Embeddings: The proposed initialization trick offers a practical way to improve performance on Chinese NLP tasks by better leveraging existing word embeddings.

Key Results

  • NER Performance: The proposed model significantly outperforms previous state-of-the-art methods on OntoNotes 5.0 English NER, achieving improvements of up to 1.41 F1 score with BERT. It also shows strong performance on Chinese NER datasets, outperforming previous work by 0.65 F1 score without external knowledge.
  • POS Tagging Performance: The model achieves competitive results on POS tagging, with notable improvements over baselines on WSJ and UD v2.2 datasets, especially when augmented with ELMo and BERT.
  • CRF Speedup: The proposed CRF implementation tricks lead to remarkable acceleration in both training and decoding speeds, making the model computationally efficient. For example, their model achieves a training speed of 219.48 sentences/sec compared to 82.25 sentences/sec for a standard BiLSTM-CRF implementation.
  • Character Embedding Initialization: The initialization trick improves Chinese NER performance by up to 0.76 F1 score on OntoNotes 4.0.

Limitations and Future Work

  • Higher-Order CRFs: The paper notes that the current implementation uses first-order CRFs, and the limited effectiveness of higher-order CRFs remains an area for future exploration.
  • Parameter Size: While their shared model is smaller than some baselines, the influence of parameter size on performance is explored, with no significant increase observed beyond a certain point.
  • Contextual Word Representations: The authors acknowledge the widespread adoption of contextual word embeddings (ELMo, BERT) and integrate them into their experiments, showing their effectiveness. However, the interaction and optimal use of these representations with their structured refinement mechanism could be further investigated.
  • Generalizability: While tested on multiple datasets, further evaluation on diverse tasks and languages could confirm the broad applicability of their structured refinement approach.

Summary updated

BibTeX
@inproceedings{wang2021structured,
  author = {Yiran Wang and Hiroyuki Shindo and Yuji Matsumoto and Taro Watanabe},
  title = {{Structured Refinement for Sequential Labeling}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
  url = {https://openreview.net/forum?id=dWN0SLxiLAb},
}

Loading related content...

Figure