DReCa: A General Task Augmentation Strategy for Few-Shot Natural Language Inference

Shikhar Murty, Tatsunori Hashimoto, Christopher D. Manning

Executive Summary

Meta-learning in NLP often fails because treating entire datasets as "tasks" leads to overfitting on small, heterogeneous data distributions. DReCa (Decomposing datasets into Reasoning Categories) addresses this by automatically clustering datasets into fine-grained, homogeneous "reasoning categories," providing a richer task distribution that improves few-shot Natural Language Inference (NLI) performance by up to 4%.

The Challenge: Meta-Learning and the "Dataset-as-Task" Trap

Meta-learning—the "learning to learn" paradigm—aims to create models that can adapt to new tasks with only a handful of examples. While successful in computer vision, it has struggled in NLP. A fundamental issue is the "task distribution" problem: in NLP, researchers typically treat an entire dataset (like MultiNLI) as a single task.

This approach suffers from two major issues:

  1. Learner Overfitting: There are only a few large-scale supervised datasets available for any given NLP problem. A meta-learner quickly memorizes the quirks of these few "tasks" rather than learning a general adaptation mechanism.
  2. Memorization Overfitting: NLP datasets are highly heterogeneous. In a random few-shot "episode," the support set might focus on negation, while the query set might focus on quantifiers. Because the support set doesn't help predict the query set, the model learns to ignore the support set entirely and just relies on its pre-trained priors.

The DReCa Method

The authors propose DReCa, a task augmentation strategy that discovers latent reasoning categories within existing datasets to create more and better tasks.

Overview of the DReCa task augmentation pipeline: (i) embedding examples, (ii) clustering into reasoning categories, and (iii) constructing tasks by pairing clusters.

The process involves four key steps:

  1. Embedding: All examples in a dataset are embedded using a BERT model finetuned on the NLI task.
  2. Label Splitting: Examples are grouped by their labels (e.g., entailment vs. not_entailment).
  3. Clustering: Within each label group, k-means clustering is applied to the BERT representations. This discovers latent "reasoning categories" such as sentences involving comparatives or coreference.
  4. Task Construction: New tasks are formed by pairing clusters from different label groups. This ensures that the support and query sets within a training episode share the same underlying reasoning type, forcing the model to use the support set for adaptation.

Latent Reasoning Categories

A key insight of the paper is that a finetuned BERT model naturally clusters linguistic phenomena in its embedding space. Even without explicit labels for "negation" or "modals," the vector space geometry separates these categories.

Examples of NLI pairs and their corresponding reasoning categories discovered or used by DReCa.

By narrowing the task to a specific reasoning type, the model is forced to find the signal in the noise. In synthetic tests (HANS), DReCa performed nearly as well as an "Oracle" model with access to true human-labeled categories, suggesting that unsupervised clustering is a highly effective proxy for expert annotation.

Key Results

The authors evaluated DReCa across several NLI benchmarks, including HANS-FEWSHOT, RTE, QANLI, and SciTail.

  • Synthetic Performance: On HANS-FEWSHOT, DReCa achieved 87.53% accuracy, significantly outperforming the standard MAML baseline (82.64%).
  • Real-World Gains: Across real NLI problems, DReCa consistently improved meta-learner accuracy:
  • +2.7 points on COMBINEDNLI-RTE.
  • +2.5 points on COMBINEDNLI-QANLI.
  • +1.6 points on GLUE-SciTail.

Limitations and Future Work

While effective, DReCa currently uses a single k-means clustering. Language is multi-faceted; a sentence might involve both "negation" and "lexical overlap." Future work could explore multi-clustering based on different factors of variation. Additionally, the strategy's dependence on the initial quality of BERT embeddings means that if the base model misses certain nuances, the clustering will too.

Summary updated

BibTeX
@inproceedings{murty2021dreca,
  author = {Shikhar Murty and Tatsunori Hashimoto and Christopher D. Manning},
  title = {{DReCa: A General Task Augmentation Strategy for Few-Shot Natural Language Inference}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
  doi = {10.18653/v1/2021.naacl-main.88},
}
Figure