Issues with Entailment-based Zero-shot Text Classification

Tingting Ma, Jin-Ge Yao, Chin-Yew Lin, Tiejun ZhaoACL 2021 · DOI

TL;DR

This paper investigates the effectiveness of using Natural Language Inference (NLI) models for zero-shot text classification. The authors reveal that this approach, while seemingly intuitive, suffers from significant issues like high variance across datasets and a surprising reliance on spurious lexical patterns, often leading to unstable and unreliable performance.

What Problem Does This Paper Address?

The general nature of Natural Language Inference (NLI) tasks, which involve determining if one sentence entails another, makes them an attractive candidate for zero-shot text classification. The idea is to frame any target label as a hypothesis and check if it's entailed by the input text. This would allow for classification across any set of labels without needing task-specific training data. However, this paper questions the practical viability of this approach, highlighting overlooked issues that hinder its effectiveness and robustness in real-world scenarios.

Methods

The authors re-examine existing work on entailment-based zero-shot text classification, particularly the approach proposed by Yin et al. (2019). They experiment with standard BERT-based NLI models fine-tuned on various NLI datasets. A key aspect of their investigation is comparing these fine-tuned models against a raw BERT model pre-trained only for "next sentence prediction" (NSP), without any NLI-specific fine-tuning. They also explore preliminary approaches to improve the robustness of NLI models against spurious lexical patterns.

Datasets

The paper uses several datasets for evaluation:

  • Topic Classification: Yahoo! Answers
  • Emotion Classification: Unify Emotion
  • Situation Classification: Situation Typing
  • Intent Detection: Snips
  • Topic Classification (News): AG's news
  • Sentiment Polarity Classification: SST-2

They also reference the HANS dataset for robustness testing.

Key Insights and Ideas

  • Raw BERT is surprisingly competitive: Pre-trained BERT models without NLI fine-tuning often perform as well as, or even better than, NLI-fine-tuned models on many zero-shot classification tasks. This suggests that the NSP pre-training might already equip BERT with useful capabilities for sentence-pair understanding relevant to classification.
  • NLI fine-tuning can be detrimental: Fine-tuning on NLI data can sometimes weaken the semantic understanding acquired during general pre-training, especially when the NLI datasets contain spurious lexical patterns.
  • High variance and instability: NLI models exhibit significant performance variance across different classification datasets. This instability suggests that they are not reliably generalizing and may require some form of in-domain data for validation to stabilize performance.
  • Reliance on spurious lexical patterns: The models appear to heavily rely on superficial word co-occurrence patterns rather than deeper semantic understanding, making them fragile and prone to errors when faced with slightly different phrasing or when the "obvious" lexical cues are absent.
  • Abstract tasks are particularly challenging: For more abstract classification tasks like emotion and situation typing, NLI models perform poorly, and fine-tuning offers little to no improvement.

Key Results

  • On several datasets (e.g., topic classification, intent classification, binary sentiment classification), raw BERT models (NSP) achieved surprisingly strong results, sometimes outperforming NLI-fine-tuned models. For instance, NSP achieved 50.6% accuracy on Topic classification (Yahoo) and 73.4% on Snips, while some NLI models performed lower.
  • The paper highlights a large variance in performance across NLI models and datasets. For example, results for the same model on different datasets could range from around 10% to over 70%.
  • Attempts to improve robustness using debiasing methods showed some gains on the HANS dataset, indicating they could mitigate reliance on word overlap heuristics. However, these methods did not lead to significant improvements on the target classification datasets.

Limitations and Future Work

The authors acknowledge that their study implies significant difficulties in building a robust zero-shot text classifier using entailment models. They question the feasibility of this approach given the current state of language technology and the challenges in collecting NLI data that promotes true out-of-domain generalization.

Future work could involve:

  • Exploring alternative schemes that leverage explicit external knowledge.
  • Developing more crafted usage of pre-trained models that better capture semantic coverage and compositionality.
  • Conducting future studies on more diverse classification scenarios to verify claimed generality.
  • Focusing on more specific task settings (e.g., zero-shot intent classification) for more reliable and stable results, potentially using more carefully curated data.

Summary updated

BibTeX
@inproceedings{ma2021issues,
  author = {Tingting Ma and Jin-Ge Yao and Chin-Yew Lin and Tiejun Zhao},
  title = {{Issues with Entailment-based Zero-shot Text Classification}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
  doi = {10.18653/v1/2021.acl-short.99},
}
Figure