Jointly Learning to Label Sentences and Tokens

Marek Rei, Anders Søgaard

Abstract

Learning to construct text representations in end-to-end systems can be difficult, as natural languages are highly compositional and task-specific annotated datasets are often limited in size. Methods for directly supervising language composition can allow us to guide the models based on existing knowledge, regularizing them towards more robust and interpretable representations. In this paper, we investigate how objectives at different granularities can be used to learn better language representations and we propose an architecture for jointly learning to label sentences and tokens. The predictions at each level are combined together using an attention mechanism, with token-level labels also acting as explicit supervision for composing sentence-level representations. Our experiments show that by learning to perform these tasks jointly on multiple levels, the model achieves substantial improvements for both sentence classification and sequence labeling.

Introduction

In Natural Language Processing (NLP), understanding text often requires analysis at multiple scales, such as determining the sentiment of an entire sentence (sentence-level) while identifying the specific words that drive that sentiment (token-level). Standard neural networks typically treat these tasks in isolation or use sentence-level classification as a "black box," relying on unsupervised attention mechanisms to select relevant words. However, these mechanisms often fail to align with human logic, particularly when training data is scarce. This research addresses two primary challenges: improving model robustness by guiding attention toward linguistically relevant tokens and leveraging sentence-level labels to improve token-level predictions in data-scarce environments.

Methodology: The BiLSTM-JOINT Architecture

The proposed BiLSTM-JOINT architecture extends the standard Bidirectional LSTM (Bi-LSTM) framework by integrating several novel components designed to synchronize sentence and token labeling.

The BiLSTM-JOINT model architecture for a single token position, illustrating how token-level predictions and weighted vector representations are derived.

Supervised Attention

Unlike traditional models where attention weights are learned implicitly, BiLSTM-JOINT treats attention weights as explicit token-level predictions. A dedicated loss function ($L_{tok}$) minimizes the discrepancy between these weights and human-provided token labels, effectively supervising the composition process.

Auxiliary Objectives

To enhance representation robustness, the model incorporates:

  • Language Modeling (LM): Simultaneous training to predict preceding and succeeding words (Word-level LM) and predicting words based on neighboring character sequences (Character-level LM).
  • Attention Range Objective ($L_{attn}$): In scenarios where token labels are missing, this objective encourages the model to maintain attention weights within reasonable bounds, such as pushing the maximum attention in a positive sentence toward 1.

Experimental Evaluation

The model was evaluated across three distinct linguistic tasks:

  1. Uncertainty Detection (CoNLL 2010): Identifying "hedging" in biomedical texts.
  2. Grammatical Error Detection (FCE): Labeling errors in essays written by English learners.
  3. Sentiment Analysis (SST): Binary classification of positive vs. negative sentiment, with tree-based annotations converted to sequence labels.

Results and Discussion

The BiLSTM-JOINT model consistently outperformed baselines that relied solely on sentence-level information (BiLSTM-LAST) or unsupervised attention (BiLSTM-ATTN).

Performance Gains

On the CoNLL 2010 dataset, the joint model improved the $F_1$ score from 83.67 to 87.17. In sentiment analysis, the inclusion of token-level supervision resulted in a 25% reduction in error. Furthermore, the model established new state-of-the-art results for error detection on the FCE and CoNLL 2014 benchmarks.

Data Efficiency and Interpretability

A key finding is the model's extreme data efficiency. As illustrated in the figure below, the joint model achieves competitive token-labeling performance even with minimal token-annotated data, effectively learning from sentence-level signals.

Sequence labeling F1 on CoNLL-10 cue detection, demonstrating the performance advantage of BiLSTM-JOINT as the percentage of token-annotated data varies.

Beyond accuracy, the model offers superior interpretability. Because attention weights are directly supervised by token labels, the model's "reasoning" for a sentence-level classification is explicitly represented as a sequence of word-level probabilities.

Conclusion and Future Work

Jointly learning sentence and token labels provides a powerful framework for improving both the accuracy and transparency of NLP models. While the current work focuses on binary classification, future research will explore extending this supervised attention framework to multi-class problems and more complex linguistic structures where the assumptions of the attention range objective may need refinement.

Summary updated

Abstract

Learning to construct text representations in end-to-end systems can be difficult, as natural languages are highly compositional and task-specific annotated datasets are often limited in size. Methods for directly supervising language composition can allow us to guide the models based on existing knowledge, regularizing them towards more robust and interpretable representations. In this paper, we investigate how objectives at different granularities can be used to learn better language representations and we propose an architecture for jointly learning to label sentences and tokens. The predictions at each level are combined together using an attention mechanism, with token-level labels also acting as explicit supervision for composing sentence-level representations. Our experiments show that by learning to perform these tasks jointly on multiple levels, the model achieves substantial improvements for both sentence classification and sequence labeling.
BibTeX
@inproceedings{rei2019jointly,
  author = {Marek Rei and Anders Søgaard},
  title = {{Jointly Learning to Label Sentences and Tokens}},
  booktitle = {AAAI},
  year = {2019},
  url = {https://openreview.net/forum?id=HQVZg1MxOpB},
}
Figure