Zero-shot Sequence Labeling: Transferring Knowledge from Sentences to Tokens

Marek Rei, Anders Søgaard

Abstract: Can attention- or gradient-based visualization techniques be used to infer token-level labels for binary sequence tagging problems, using networks trained only on sentence-level labels? We construct a neural network architecture based on soft attention, train it as a binary sentence classifier and evaluate against token-level annotation on four different datasets. Inferring token labels from a network provides a method for quantitatively evaluating what the model is learning, along with generating useful feedback in assistance systems. Our results indicate that attention-based methods are able to predict token-level labels more accurately, compared to gradient-based methods, sometimes even rivaling the supervised oracle network.

Introduction

Sequence labeling—the task of assigning a label to every word in a sentence—is a cornerstone of Natural Language Processing (NLP), utilized in applications ranging from part-of-speech tagging to named entity recognition. However, a significant bottleneck exists: the cost of data acquisition. Manually labeling every token in a massive dataset is prohibitively expensive and time-consuming.

In contrast, sentence-level labels (e.g., "this sentence contains a grammatical error") are much easier to obtain. This research explores a zero-shot sequence labeling approach, asking whether a model can be trained on coarse sentence labels and successfully "figure out" which specific tokens triggered those labels. This bridges the gap between high-level classification and fine-grained sequence tagging without requiring token-level supervision during training.

Methods

The authors propose a neural architecture centered on a Bidirectional Long Short-Term Memory (Bi-LSTM) network enhanced by a novel attention mechanism.

Architecture and Attention

As shown in Figure 1, tokens are first converted into representations using both word and character-level embeddings. These are processed by a Bi-LSTM to capture bidirectional context.

Figure 1: Neural architecture for zero-shot sequence labeling featuring Bi-LSTM and sigmoid-based attention.

The critical innovation is the "soft attention" layer. Standard attention mechanisms typically utilize a softmax function, which forces the model to distribute a fixed probability mass, effectively picking a single "winner." This is ill-suited for sequence labeling where a sentence might contain zero or multiple relevant tokens. Instead, the authors employ a sigmoid-based soft attention, allowing each word to independently receive a score between 0 and 1.

Auxiliary Loss Functions

To ensure attention weights behave like binary labels, two specialized constraints were introduced:

  1. Sparsity ($L_2$): Encourages the minimum attention weight in a sentence to be 0, reflecting that not every word is relevant.
  2. Consistency ($L_3$): Forces the maximum attention weight to match the sentence label; if a sentence is labeled positive, at least one word must have a high attention score.

During inference, any token with an attention weight exceeding 0.5 is predicted as a positive token.

Results

The model was evaluated across three distinct tasks: uncertainty detection (CoNLL 2010), grammatical error detection (FCE), and sentiment detection (SemEval). The attention-based method (LSTM-ATTN-SW) was compared against gradient-based backpropagation (saliency maps) and a relative frequency baseline.

Table 1: Performance comparison on CoNLL 2010 and FCE datasets showing the superiority of the attention-based zero-shot approach.

As detailed in Table 1, the attention-based method consistently outperformed the alternatives. In the CoNLL 2010 task, the zero-shot model achieved a token-level $F_1$ score of 73.26%, reaching 93% of the performance of a fully supervised "oracle" model (79.08%). While the FCE error detection task proved more difficult due to its contextual nature, the zero-shot model still achieved an $F_1$ of 28.27%, significantly higher than the gradient-based approach (21.65%).

Discussion and Conclusion

This work demonstrates that attention mechanisms can transition from performance boosters to quantitative evaluation tools. By treating attention weights as labels, researchers can measure how well a model's internal focus aligns with human intuition. The shift from softmax to sigmoid attention, combined with mathematically defined label properties ($L_2$ and $L_3$), allows for effective knowledge transfer from sentence-level supervision to token-level inference.

Future work will focus on improving performance for purely contextual errors (e.g., missing words) and extending the framework to multi-class zero-shot labeling to handle conflicting signals within a single sentence.

Summary updated

Abstract

Can attention- or gradient-based visualization techniques be used to infer token-level labels for binary sequence tagging problems, using networks trained only on sentence-level labels? We construct a neural network architecture based on soft attention, train it as a binary sentence classifier and evaluate against token-level annotation on four different datasets. Inferring token labels from a network provides a method for quantitatively evaluating what the model is learning, along with generating useful feedback in assistance systems. Our results indicate that attention-based methods are able to predict token-level labels more accurately, compared to gradient-based methods, sometimes even rivaling the supervised oracle network.
BibTeX
@inproceedings{rei2018zero,
  author = {Marek Rei and Anders Søgaard},
  title = {{Zero-shot Sequence Labeling: Transferring Knowledge from Sentences to Tokens}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2018},
  url = {https://openreview.net/forum?id=SJZmtQWObB},
}
Figure