On Measuring the Intrinsic Few-Shot Hardness of Datasets

Xinran Zhao, Shikhar Murty, Christopher D. Manning

Introduction

Few-shot learning—the ability of a model to adapt to a new task using only a handful of examples—has become a cornerstone of modern Natural Language Processing (NLP). While massive pre-trained models like RoBERTa or GPT variants show remarkable adaptability, their performance is notoriously inconsistent across different tasks. A model might achieve near-perfect accuracy on sentiment analysis with five examples but fail completely on complex paraphrasing tasks. This inconsistency raises a fundamental question: is few-shot performance a result of the specific adaptation algorithm, or is it an intrinsic property of the dataset itself?

This paper investigates the concept of Intrinsic Few-Shot Hardness (IFH). The authors demonstrate that the difficulty of a task is largely independent of the adaptation method used (e.g., prompting vs. fine-tuning) and propose Spread, a lightweight metric that predicts this hardness by analyzing the model's latent feature space.

Defining Intrinsic Few-Shot Hardness

To determine if hardness is intrinsic, the researchers evaluated several state-of-the-art adaptation methods across diverse tasks. They found a high Spearman correlation (0.85) between the performance of different methods, such as prompt-based learning and lightweight fine-tuning. If a dataset is "hard" for one method, it is consistently hard for others. This suggests that "few-shot learnability" is a property of the dataset-model pair rather than a byproduct of the specific optimization strategy.

The Spread Metric: Measuring Feature-Space Invariance

The authors hypothesize that few-shot success depends on how well the training examples "cover" the test set within the model's internal representation. If a test example's features are close to a training example's features in the latent space, the model can generalize more effectively.

To quantify this, they introduce Spread. Spread calculates the average Euclidean distance from each test input feature to its nearest neighbor in the training set within the same class. Unlike previous metrics, Spread does not require expensive training loops or multiple iterations of fine-tuning.

Comparison with Existing Metrics

The researchers compared Spread against two established benchmarks:

  1. Sensitivity Analysis (SA): Measures how much a model's prediction changes in response to small input perturbations.
  2. Rissanen Data Analysis (RDA): Measures hardness based on the area under the test loss curve during the training process.

Experimental Setup and Results

The study utilized two primary benchmarks: FS-GLUE (a few-shot version of 11 standard GLUE/SuperGLUE tasks) and FS-NLI (28 datasets recast into a uniform Natural Language Inference format, covering tasks from sentiment analysis to named entity classification).

Predictive Accuracy and Efficiency

Spread outperformed existing metrics in predicting actual few-shot hardness, achieving a correlation of 0.47, compared to 0.36 for RDA and a negligible 0.06 for SA. Furthermore, Spread is significantly more efficient, running 8x to 100x faster than RDA because it bypasses the need for multiple rounds of model training.

Dataset Decomposition and Performance Gains

Beyond measurement, Spread offers actionable insights for data selection. By clustering a dataset and sampling training points that minimize Spread (ensuring better coverage of the feature space), the authors achieved an average accuracy boost of 13.9 points on named entity classification tasks compared to random sampling.

Key Insights

  • Model Independence: Interestingly, tasks that are difficult for RoBERTa are often similarly difficult for ELECTRA. This suggests that certain factors making a task "hard" for transformers may be independent of the specific pre-trained architecture.
  • Feature Proximity is Key: The core driver of few-shot success is the alignment between training and test distributions in the latent space. When training examples are outliers relative to the test set, the model fails to generalize.

Limitations and Future Directions

While this work provides a robust framework for dataset-level hardness, several avenues remain for future research:

  • Granularity: The current study focuses on dataset-level hardness; investigating why specific individual instances are difficult remains an open challenge.
  • Model Scale: Future work should explore whether these findings hold for non-transformer architectures or extremely large-scale models like GPT-4.
  • Pre-training Influence: The relationship between a model's original pre-training data and its few-shot performance on specific downstream tasks warrants further investigation to determine if "hardness" is a reflection of data gaps during initial training.

Summary updated

BibTeX
@inproceedings{zhao2022measuring,
  author = {Xinran Zhao and Shikhar Murty and Christopher D. Manning},
  title = {{On Measuring the Intrinsic Few-Shot Hardness of Datasets}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  doi = {10.18653/v1/2022.emnlp-main.262},
}
Figure