Locally-Contextual Nonlinear CRFs for Sequence Labeling

Harshil Shah, Tim Xiao, David BarberarXiv 2021 · OpenReview

TL;DR

This paper introduces a novel sequence labeling model, the Locally-Contextual Nonlinear CRF (CRF®), that directly leverages neighboring word embeddings to improve predictions. By replacing linear potential functions with deep neural networks, the CRF® achieves state-of-the-art results on several benchmark tasks, outperforming previous methods, especially when using powerful contextual embeddings like BERT and Flair.

What Problem Does This Paper Address?

Sequence labeling tasks, such as part-of-speech tagging or named entity recognition, involve assigning a label to each word in a sequence. Traditional methods like linear chain CRFs model dependencies between adjacent labels but don't explicitly use information from surrounding words (context) when predicting a label for a specific word. While modern contextual word embeddings (like BERT) capture sentence-level context, they are trained in a general-purpose way and don't guarantee that the most relevant contextual information for sequence labeling is encoded. This paper addresses the gap by proposing a model that explicitly extracts and utilizes local contextual information from word embeddings, improving prediction accuracy.

Methods

The core innovation is the Locally-Contextual Nonlinear CRF (CRF®). It builds upon the standard linear chain CRF by:

  • Directly Incorporating Local Context: Instead of relying solely on the embedding of the current word (ht) to implicitly capture neighboring information, the CRF® explicitly uses embeddings of adjacent words (ht-1 and ht+1) when modeling the potential function for the current label (yt). This is achieved by introducing new "local context" potentials (φt and ξt).
  • Nonlinear Potential Functions: The linear potential functions of traditional CRFs are replaced with deep neural networks. This allows for more flexible and powerful modeling of the relationships between labels and contextual information.

The CRF® is designed as a "drop-in replacement" for linear chain CRFs, meaning it can be integrated into existing pipelines with minimal changes. The paper also explores variations like using bidirectional LSTMs with the CRF® and widening the local context window, though these did not consistently yield improvements over the core CRF® design.

Datasets

The proposed model was evaluated on several standard sequence labeling datasets:

  • Chunking: CoNLL 2000
  • Part-of-Speech Tagging: Penn Treebank
  • Named Entity Recognition: CoNLL 2003 and OntoNotes 5.0 English

The paper does not introduce new benchmarks but uses established datasets with standard train/validation/test splits.

Key Insights and Ideas

  • Explicit Local Context is Crucial: Even with powerful contextual embeddings like BERT, explicitly feeding neighboring word embeddings into the model's potential functions provides significant gains. This suggests that task-agnostic embeddings might not always perfectly capture the most salient local context for sequence labeling.
  • Nonlinearity Matters: Replacing linear potential functions with deep neural networks offers greater modeling flexibility and leads to improved performance.
  • BERT and Flair Embeddings Shine: The CRF® architecture is particularly effective when paired with advanced contextual embeddings like BERT and Flair, achieving state-of-the-art results. This highlights the synergy between sophisticated embeddings and tailored model architectures.
  • Bidirectional LSTMs Not Always Necessary with Strong Embeddings: When using BERT or Flair embeddings, adding a bidirectional LSTM layer did not consistently improve performance. This is likely because these embeddings already encode rich contextual information, making the additional LSTM redundant.

Key Results

The CRF® consistently outperformed its linear counterparts and previous state-of-the-art methods across the evaluated tasks:

  • Chunking (CoNLL 2000): The CRF® (especially with BERT and Flair embeddings) outperformed the previous state-of-the-art. For example, CRF®(Flair) achieved an F1 score of 97.52, surpassing the previous best of 97.30.
  • Named Entity Recognition (OntoNotes 5.0 English): CRF®(BERT) achieved an F1 score of 92.17, outperforming the previous state-of-the-art.
  • Ablation Studies: The paper demonstrated that both the local context and the nonlinear potentials individually contributed to performance improvements, with their combination (CRF®) yielding the best results.
  • Computational Efficiency: The CRF® is approximately 1.5-2 times slower for training and inference than a standard linear CRF, but this trade-off is justified by the performance gains.

Limitations and Future Work

  • Computational Cost: The CRF® is computationally more expensive than linear CRFs due to the added local context potentials and nonlinear functions.
  • Joint Training with Embeddings: The authors suggest that jointly training contextual embedding models (like BERT) with sequence labeling tasks could potentially eliminate the need for explicit local contextual connections in the CRF®. This is identified as a promising avenue for future research.
  • External Knowledge Integration: The paper notes that the CRF® could benefit from being augmented with external knowledge bases, which has shown promise in improving named entity recognition performance.

Summary updated

Abstract

Linear chain conditional random fields (CRFs) combined with contextual word embeddings have achieved state of the art performance on sequence labeling tasks. In many of these tasks, the identity of the neighboring words is often the most useful contextual information when predicting the label of a given word. However, contextual embeddings are usually trained in a task-agnostic manner. This means that although they may encode information about the neighboring words, it is not guaranteed. It can therefore be beneficial to design the sequence labeling architecture to directly extract this information from the embeddings. We propose locally-contextual nonlinear CRFs for sequence labeling. Our approach directly incorporates information from the neighboring embeddings when predicting the label for a given word, and parametrizes the potential functions using deep neural networks. Our model serves as a drop-in replacement for the linear chain CRF, consistently outperforming it in our ablation study. On a variety of tasks, our results are competitive with those of the best published methods. In particular, we outperform the previous state of the art on chunking on CoNLL 2000 and named entity recognition on OntoNotes 5.0 English.
BibTeX
@misc{shah2021locally,
  author = {Harshil Shah and Tim Xiao and David Barber},
  title = {{Locally-Contextual Nonlinear CRFs for Sequence Labeling}},
  year = {2021},
  eprint = {2103.16210},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2103.16210},
}
Figure