An Exploration of Arbitrary-Order Sequence Labeling via Energy-Based Inference Networks

Lifu Tu, Tianyu Liu, Kevin Gimpel

Abstract

Many tasks in natural language processing involve predicting structured outputs, e.g., sequence labeling, semantic role labeling, parsing, and machine translation. Researchers are increasingly applying deep representation learning to these problems, but the structured component of these approaches is usually quite simplistic. In this work, we propose several high-order energy terms to capture complex dependencies among labels in sequence labeling, including several that consider the entire label sequence. We use neural parameterizations for these energy terms, drawing from convolutional, recurrent, and self-attention networks. We use the framework of learning energy-based inference networks (Tu and Gimpel, 2018) for dealing with the difficulties of training and inference with such models. We empirically demonstrate that this approach achieves substantial improvement using a variety of high-order energy terms on four sequence labeling tasks, while having the same decoding speed as simple, local classifiers. We also find high-order energies to help in noisy data conditions.

Introduction

Sequence labeling tasks, such as Part-of-Speech (POS) tagging and Named Entity Recognition (NER), are fundamental to natural language processing. Traditionally, these tasks rely on Conditional Random Fields (CRFs) to model dependencies between labels. However, standard CRFs are typically "first-order," restricted to modeling relationships between immediate neighbors. While higher-order dependencies could capture richer structural patterns, their computational cost grows exponentially with the order of the dependency, making exact inference prohibitively slow. Consequently, most practical systems default to local models, sacrificing the benefits of long-range structural information.

Methodology: Energy-Based Inference Networks

To overcome these limitations, we utilize the framework of Structured Prediction Energy Networks (SPENs). We define an energy function $E(x, y)$ that scores the compatibility between an input sequence $x$ and a label sequence $y$. Our goal is to find the sequence $y$ that minimizes this energy.

We introduce several concise neural parameterizations to capture arbitrary-order dependencies:

  • CNNs: Use filters to detect local label patterns like trigrams.
  • Self-Attention: Models global dependencies across the entire sequence.
  • Vectorized Kronecker Products (VKP): Provides a memory-efficient representation for high-order tensors.
  • Tag Language Models (TLM): Scores sequences based on their likelihood under a pre-trained label model.

To maintain efficiency, we employ inference networks—separate neural networks trained to approximate the energy minimizer. This approach allows us to leverage complex high-order logic during training while ensuring that test-time decoding remains as fast as a simple local classifier.

Complexity comparison of different modeling approaches.

Experimental Evaluation

We evaluated our approach on four benchmarks: Twitter POS tagging, CoNLL 2003 (NER), CCG-bank supertagging, and CoNLL 2005 Semantic Role Labeling (SRL). Our models consistently outperformed BiLSTM-CRF baselines. For instance, on NER, our high-order models achieved a gain of approximately 0.8 F1 points over linear-chain CRFs. Furthermore, these energy terms proved complementary to large pre-trained models; adding a skip-chain energy term to a BERT-based model improved NER performance from 92.13 to 92.46 F1.

Robustness to Noise

A key finding of this work is the robustness of high-order models in noisy data conditions. When input text is degraded (e.g., by replacing words with "UNK" tokens), local context becomes unreliable. In these scenarios, the model relies on the global "shape" of the label sequence captured by high-order energies.

Performance comparison under varying levels of data noise.

In experiments where 30% of words were replaced with noise, our CNN-based high-order model outperformed local classifiers by 2.0 points and linear CRFs by 1.8 points. This demonstrates that structural intuition is particularly valuable when local signals are weak.

Conclusion

By combining arbitrary-order energy functions with efficient inference networks, we demonstrate that it is possible to capture complex structural dependencies without sacrificing decoding speed. Different tasks benefit from different architectures—for example, SRL benefits from the long-range modeling of self-attention. Future work will explore extending this framework to even more complex output spaces, such as dependency parsing and machine translation.

Summary updated

Abstract

Many tasks in natural language processing involve predicting structured outputs, e.g., sequence labeling, semantic role labeling, parsing, and machine translation. Researchers are increasingly applying deep representation learning to these problems, but the structured component of these approaches is usually quite simplistic. In this work, we propose several high-order energy terms to capture complex dependencies among labels in sequence labeling, including several that consider the entire label sequence. We use neural parameterizations for these energy terms, drawing from convolutional, recurrent, and self-attention networks. We use the framework of learning energy-based inference networks (Tu and Gimpel, 2018) for dealing with the difficulties of training and inference with such models. We empirically demonstrate that this approach achieves substantial improvement using a variety of high-order energy terms on four sequence labeling tasks, while having the same decoding speed as simple, local classifiers. We also find high-order energies to help in noisy data conditions.
BibTeX
@inproceedings{tu2020exploration,
  author = {Lifu Tu and Tianyu Liu and Kevin Gimpel},
  title = {{An Exploration of Arbitrary-Order Sequence Labeling via Energy-Based Inference Networks}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  url = {https://openreview.net/forum?id=GZTRQWMK8g},
}
Figure