Zero-Shot Sequence Labeling: Transferring Knowledge from Sentences to Tokens

Marek Rei, Anders Søgaard

Abstract

Can attention- or gradient-based visualization techniques be used to infer token-level labels for binary sequence tagging problems, using networks trained only on sentence-level labels? We construct a neural network architecture based on soft attention, train it as a binary sentence classifier and evaluate against token-level annotation on four different datasets. Inferring token labels from a network provides a method for quantitatively evaluating what the model is learning, along with generating useful feedback in assistance systems. Our results indicate that attention-based methods are able to predict token-level labels more accurately, compared to gradient-based methods, sometimes even rivaling the supervised oracle network.

Introduction

Sequence labeling is a fundamental task in Natural Language Processing (NLP), requiring the assignment of labels to individual tokens within a sentence. Applications range from Named Entity Recognition to Grammatical Error Detection. However, obtaining token-level annotations is labor-intensive and expensive. In contrast, sentence-level labels (e.g., sentiment polarity or the presence of an error) are significantly easier to acquire. This research investigates whether a model trained exclusively on sentence-level labels can successfully infer token-level labels—a "zero-shot" approach to sequence labeling.

Methodology

The proposed architecture utilizes a Bidirectional Long Short-Term Memory (Bi-LSTM) network to capture contextual information. The core innovation lies in the attention mechanism used to aggregate token-level representations into a sentence-level vector.

The neural network architecture featuring a Bi-LSTM and a specialized attention mechanism for zero-shot inference.

Unlike standard attention that often employs a softmax function—forcing the model to focus on a single dominant token—this approach uses a sigmoid-based attention normalized to sum to 1. This "collaborative" attention allows the model to highlight multiple relevant tokens simultaneously. To ensure the attention weights ($a_i$) function effectively as token labels, two auxiliary loss constraints were introduced:

  1. Sparsity ($L_2$): Encourages the minimum attention weight in a sentence to be zero, assuming not all words are relevant to the label.
  2. Consistency ($L_3$): Ensures the maximum attention weight aligns with the sentence-level gold label.

Results

The model was evaluated across four datasets: CoNLL 2010 (Uncertainty Detection), FCE (Error Detection), and SemEval Sentiment (Positive and Negative). The attention-based method (LSTM-ATTN-SW) consistently outperformed gradient-based visualization and statistical baselines.

Quantitative results comparing the zero-shot attention model against supervised and baseline methods across CoNLL 2010 and FCE datasets.

In the CoNLL 2010 task, the zero-shot model achieved an F1 score of 73.26, reaching approximately 93% of the performance of the fully supervised oracle (79.08 F1). While the FCE Error Detection task proved more challenging due to the complexity of grammatical errors, the model still demonstrated the ability to identify misspellings and punctuation issues without explicit token-level training. Qualitative analysis further revealed that the model could correctly identify multi-word cues and subtle errors, such as missing punctuation.

Discussion and Conclusion

This work demonstrates that attention is not merely a visualization tool but a functional component capable of quantitative zero-shot learning. The sigmoid-based attention mechanism proves superior for sequence labeling by allowing for multi-token relevance. While currently limited to binary tasks and susceptible to "negative evidence" (where the model attends to words that suggest the absence of a class), this framework offers a promising path toward interpretable-by-design models that leverage coarse supervision for granular tasks.

Summary updated

Abstract

Can attention- or gradient-based visualization techniques be used to infer token-level labels for binary sequence tagging problems, using networks trained only on sentence-level labels? We construct a neural network architecture based on soft attention, train it as a binary sentence classifier and evaluate against token-level annotation on four different datasets. Inferring token labels from a network provides a method for quantitatively evaluating what the model is learning, along with generating useful feedback in assistance systems. Our results indicate that attention-based methods are able to predict token-level labels more accurately, compared to gradient-based methods, sometimes even rivaling the supervised oracle network.
BibTeX
@inproceedings{rei2018zero,
  author = {Marek Rei and Anders Søgaard},
  title = {{Zero-Shot Sequence Labeling: Transferring Knowledge from Sentences to Tokens}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2018},
  url = {https://openreview.net/forum?id=SJZmtQWObB},
}
Figure