Joint Training with Semantic Role Labeling for Better Generalization in Natural Language Inference

Cemil Cengiz, Deniz YuretACL 2020

TL;DR

This paper tackles the problem of Natural Language Inference (NLI) models failing to generalize to out-of-distribution datasets, often relying on superficial heuristics. They propose a novel approach of jointly training a BERT model on both NLI and Semantic Role Labeling (SRL) tasks, showing significant improvements in generalization without sacrificing performance on the original tasks.

What Problem Does This Paper Address?

Natural Language Inference (NLI) is a fundamental task in NLP, aiming to determine the relationship (entailment, contradiction, neutral) between two sentences. While models like BERT achieve state-of-the-art performance on standard NLI benchmarks (e.g., MultiNLI, SNLI), they often fail when presented with adversarial examples designed to expose dataset biases. These models tend to learn shallow "shortcuts" or heuristics, such as relying heavily on lexical overlap between sentences, rather than truly understanding the meaning. This leads to poor performance on specialized evaluation sets like HANS, which are specifically designed to test robustness against these heuristics. Existing solutions, like data augmentation, can be brittle and may even hurt performance on unseen adversaries.

Methods

The core of this paper's contribution is a multi-task learning (MTL) framework that jointly trains a BERT model on both NLI and Semantic Role Labeling (SRL).

  • Shared Encoder: A pre-trained BERT model serves as the shared encoder for both tasks. This allows the model to learn rich contextual representations that can be leveraged by both NLI and SRL.
  • Task-Specific Decoders: Separate "decoder heads" are appended to the BERT encoder for each task.
  • For NLI, a standard classification head predicts the entailment, contradiction, or neutral label.
  • For SRL, a sequence tagging head predicts the semantic roles of words (e.g., who did what to whom).
  • Joint Training: The model is trained simultaneously on both NLI and SRL datasets. The loss from both tasks is combined, and the BERT encoder's weights are updated to optimize performance across both objectives. The hypothesis is that by forcing the model to explicitly identify semantic roles, it will develop a deeper understanding of sentence structure and meaning, making it less susceptible to superficial heuristics in NLI.

Datasets

  • Training Datasets:
    • MultiNLI: A large-scale, diverse NLI dataset with five genres of written and spoken English.
  • English Ontonotes v5.0 SRL: A dataset providing predicate-argument structures for sentences, used for the SRL task.
  • Evaluation Datasets:
    • HANS (Heuristic Analysis for NLI Systems): An adversarial NLI dataset designed to test robustness against common heuristics like lexical overlap and subsequence. It consists of three types of adversarial examples: lexical overlap, subsequence, and constituent.
  • Comparisons: Another adversarial NLI dataset designed to evaluate models on specific types of sentence pairs (same, more-less, not) that can expose heuristic reliance.

Key Insights and Ideas

  • SRL as an Inductive Bias for NLI: The paper posits that explicitly training a model to understand semantic roles acts as a powerful inductive bias for NLI. This forces the model to go beyond surface-level patterns and focus on the underlying meaning and relationships between sentence components.
  • Generalization vs. Specific Adversary Training: The authors argue that their MTL approach is more advantageous than traditional data augmentation for adversarial examples. While augmentation might improve performance on specific, targeted adversaries, it risks overfitting and can even degrade performance on other, unseen adversarial patterns. The MTL approach, by fostering a deeper semantic understanding, is expected to generalize better across a wider range of adversaries.
  • Shared Representations are Key: The success of the MTL approach hinges on the BERT encoder learning representations that are beneficial for both SRL and NLI. This shared representation space is hypothesized to be more robust and semantically rich.

Key Results

  • Significant Improvement on Adversarial Datasets: The multi-task BERT model achieved significantly higher accuracy on HANS and Comparisons compared to a single-task BERT model trained only on NLI.
  • On HANS, the multi-task model exceeded the single-task model accuracy by 4% without data augmentation.
  • On Comparisons, the multi-task model showed a 5.3% improvement when trained on MultiNLI.
  • Robustness to Heuristics: The multi-task model demonstrated improved performance on non-entailment examples in HANS, particularly in the lexical overlap category (e.g., passive voice examples), where the single-task model struggled. This indicates a better ability to discern meaning changes even when lexical overlap is high.
  • Comparable Performance on Original Tasks: The joint training did not negatively impact performance on the original NLI tasks. The multi-task model achieved accuracy comparable to the single-task model on standard NLI validation sets.
  • SRL Performance Maintained: The F1 score on the SRL test dataset for the multi-task model was 86.0, which is comparable to state-of-the-art single-task SRL models, indicating that joint training did not harm the SRL performance.

Limitations and Future Work

  • Sequential Transfer Learning Ineffectiveness: The paper explored sequential transfer learning (SRL first, then NLI) and found it less effective than MTL. They hypothesize this is because the SRL knowledge might be "forgotten" during the subsequent NLI training.
  • Future Work: The authors suggest incorporating joint prediction of all predicates and their corresponding roles to further analyze its effect on adversarial NLI performance. They also propose exploring other semantic or syntactic tasks as auxiliary objectives.

Summary updated

BibTeX
@inproceedings{cengiz2020joint,
  author = {Cemil Cengiz and Deniz Yuret},
  title = {{Joint Training with Semantic Role Labeling for Better Generalization in Natural Language Inference}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
}
Figure