Jointly Learning to Label Sentences and Tokens.

Marek Rei, Anders Søgaard

Abstract

Learning to construct text representations in end-to-end systems can be difficult, as natural languages are highly compositional and task-specific annotated datasets are often limited in size. Methods for directly supervising language composition can allow us to guide the models based on existing knowledge, regularizing them towards more robust and interpretable representations. In this paper, we investigate how objectives at different granularities can be used to learn better language representations and we propose an architecture for jointly learning to label sentences and tokens. The predictions at each level are combined together using an attention mechanism, with token-level labels also acting as explicit supervision for composing sentence-level representations. Our experiments show that by learning to perform these tasks jointly on multiple levels, the model achieves substantial improvements for both sentence classification and sequence labeling.

Introduction

In Natural Language Processing (NLP), researchers often seek to determine both the overall category of a sentence (classification) and the specific words that trigger that category (sequence labeling). Standard end-to-end models typically focus on one of these tasks. In sentence classification, attention mechanisms are often learned in an unsupervised manner, where the model guesses word importance to minimize error. Conversely, sequence labelers often ignore the broader sentence context.

This paper addresses the gap caused by scarce human-annotated data. The authors propose that by supervising the attention mechanism with token-level labels, models can be guided to focus on the same features a human would, leading to improved accuracy and higher interpretability.

Methodology

The core of this research is the BiLSTM-JOINT architecture, which integrates multiple levels of linguistic information to learn robust representations.

The BiLSTM-JOINT architecture diagram showing the flow from character/word embeddings to joint sentence and token predictions.

The model is composed of several key layers:

  1. Input Layer: Combines pre-trained word embeddings with character-level representations (via a Bi-LSTM) to capture morphological features.
  2. Context Layer: A word-level Bi-LSTM processes the sequence to create context-aware hidden states ($h_i$) for each token.
  3. The Joint Mechanism: This is the central innovation. The model predicts a score ($\hat{a}_i$) for every token. In sequence labeling, this score is the predicted label. In sentence classification, these same scores are normalized and used as attention weights to pool word representations into a single sentence vector ($s$).
  4. Multi-task Objectives: The model is optimized using a weighted sum of five loss functions, including standard sentence/token errors, auxiliary language modeling objectives ($L_{LM}$ & $L_{char}$), and a novel Attention Range Objective ($L_{attn}$) that encourages salient attention even when token labels are missing.

Experimental Results

The model was evaluated on three distinct tasks: hedging detection (CoNLL 2010), grammatical error detection (FCE), and sentiment analysis (SST).

Performance comparison of BiLSTM-JOINT against baselines on CoNLL 2010 and FCE datasets.

Key insights from the experiments include:

  • Performance Gains: The BiLSTM-JOINT model consistently outperformed baselines across all datasets. As shown in the table above, it achieved significant improvements in $F_1$ scores for both CoNLL 2010 and FCE compared to standard Bi-LSTM models with unsupervised attention.
  • Supervised Attention: Explicitly telling the model which words are important via token labels significantly improved sentence-level accuracy, leading to a 25% reduction in error for sentiment analysis.
  • Data Efficiency: The joint model proved highly efficient; with only 20% of token-level annotations, it matched the performance of a standard sequence labeler trained on 100% of the data.
  • Mutual Benefit: The two tasks provide reciprocal benefits: sentence-level labels act as a global constraint for the token labeler, while token labels provide the ground truth for the composition function.

Conclusion

Jointly learning to label sentences and tokens provides a powerful framework for improving NLP model performance and interpretability. By supervising attention mechanisms, the model learns to compose representations that align more closely with human judgment. Future work includes extending this architecture to multi-class problems and applying the framework to hierarchical document-level classification.

Summary updated

Abstract

Learning to construct text representations in end-to-end systems can be difficult, as natural languages are highly compositional and task-specific annotated datasets are often limited in size. Methods for directly supervising language composition can allow us to guide the models based on existing knowledge, regularizing them towards more robust and interpretable representations. In this paper, we investigate how objectives at different granularities can be used to learn better language representations and we propose an architecture for jointly learning to label sentences and tokens. The predictions at each level are combined together using an attention mechanism, with token-level labels also acting as explicit supervision for composing sentence-level representations. Our experiments show that by learning to perform these tasks jointly on multiple levels, the model achieves substantial improvements for both sentence classification and sequence labeling.
BibTeX
@inproceedings{rei2019jointly,
  author = {Marek Rei and Anders Søgaard},
  title = {{Jointly Learning to Label Sentences and Tokens.}},
  booktitle = {AAAI},
  year = {2019},
  url = {https://openreview.net/forum?id=HQVZg1MxOpB},
}
Figure