Learning New Tasks from a Few Examples with Soft-Label Prototypes

Avyav Singh, Ekaterina Shutova, Helen YannakoudakisACL 2024

TL;DR

This paper introduces Soft-Label Prototypes (SLPs), a novel and parameter-efficient approach for few-shot learning in NLP. SLPs allow models to learn new tasks with very few examples by capturing class distributions, achieving state-of-the-art results in limited-resource settings and competitive performance in high-resource settings.

What Problem Does This Paper Address?

Few-shot learning in Natural Language Processing (NLP) aims to enable models to learn new tasks with minimal training data, mimicking human adaptability. Existing methods often rely on large language models (LLMs) and extensive fine-tuning, which can be computationally expensive and lack standardization in prompting. Furthermore, evaluating these models on unseen tasks (out-of-distribution generalization) is challenging due to potential data leakage and the vastness of LLM training data. This paper addresses the need for a more efficient, standardized, and robust few-shot learning approach that performs well even with extremely limited data.

Methods

The core innovation is Soft-Label Prototypes (SLPs). Unlike traditional hard labels (one-hot encoded), soft labels represent a data point's simultaneous membership to multiple classes, essentially capturing a probabilistic distribution.

The authors propose two main approaches:

  • DeepSLP: This is a parameter-efficient framework that learns SLPs. It involves:
  • Finding lines connecting class centroids: This step groups similar classes based on their data distribution.
  • Learning SLPs via gradient descent: Two trainable linear layers are used to generate soft-label probability distributions at the endpoints of these lines. Crucially, the encoder (BERT in this case) is kept frozen, making the approach computationally light.
  • MetaSLP: This is an extension of DeepSLP for high-resource settings. It incorporates meta-learning, allowing the encoder to be fine-tuned using auxiliary training data. This approach aims to improve generalization across a wider range of tasks.

The classification process involves calculating the distance of a test point to the nearest SLPs and weighing their soft labels accordingly.

Datasets

The authors evaluate their methods on a variety of datasets, categorized into:

  • Limited-resource setting: This setting uses only the few-shot fine-tuning data for unseen tasks, without auxiliary training data.
  • High-resource setting: This setting utilizes auxiliary training data for meta-learning.

The datasets used include:

  • GLUE benchmark: A collection of natural language tasks (MNLI, SST2, COLA, MRPC, QQP, QNLI, RTE, SNLI) used for meta-training in the high-resource setting.
  • Amazon Review Data: Used for sentiment classification (music, toys, videos).
  • Entity Typing tasks: CoNLL-2003 and MIT-Restaurant datasets.
  • Review rating classification: A three-way classification task based on Amazon Reviews.
  • Text classification: Scraped social media data for sentiment, emotion, and political bias detection.
  • Natural Language Inference: SciTail dataset.

The authors use the same data splits and evaluation settings as Bansal et al. (2020a) for comparability.

Key Insights and Ideas

  • Soft Labels as Class Distributions: The core idea of representing classes not as discrete entities but as probability distributions over data points is powerful for few-shot learning, allowing for more nuanced representation.
  • Parameter Efficiency: DeepSLP achieves state-of-the-art results with a significantly smaller number of trainable parameters compared to LLM fine-tuning methods, making it more accessible and faster.
  • Line-based Prototypes: Connecting class centroids with lines and using their endpoints as prototypes provides a structured way to capture inter-class relationships.
  • ConstraintSLP vs. DeepSLP: While constraint-based SLPs (constraintSLP) offer a simpler approach, DeepSLP's gradient-based learning of soft labels proves more flexible and robust, especially for tasks with more than two classes. ConstraintSLP can exhibit erratic behavior and fail to select the closest centroid.
  • Ensemble-like Properties: DeepSLP exhibits properties similar to ensembles, such as higher uncertainty driven by diversity, which contributes to its robustness and performance, without the computational overhead of traditional ensembles.

Key Results

  • Low-Resource Setting: DeepSLP outperforms all baselines, including BERT fine-tuned and LORABERT, on the majority of tasks (31/48 for BERT fine-tuned, 45/48 for LORABERT), achieving new state-of-the-art results.
  • High-Resource Setting: MetaSLPREPTILE achieves the highest performance overall in text classification and entailment tasks (33/42 tasks/settings), outperforming LEOPARD and other baselines.
  • Parameter Efficiency: DeepSLP requires significantly fewer trainable parameters (1500-10K) compared to LORABERT (50K-100K) and BERT fine-tuned (>10⁸).
  • Robustness: DeepSLP shows improved performance and stability compared to constraintSLP, particularly for multi-class scenarios.

Limitations and Future Work

  • Data Contamination: The authors explicitly state that they use BERT as their encoder because it has not been pre-trained on the test data, avoiding data contamination issues that can arise with more recent LLMs.
  • Meta-training Data Distribution: In the high-resource setting, performance on the "Disaster" task was lower than in the low-resource setting. The authors surmise this is due to the auxiliary data's meta-training distribution differing significantly from the test distribution.
  • Future Work: The authors plan to explore meta-learning algorithms like PACMAML and Bayesian MAML to relax assumptions about train-test set distributions, further alleviating current limitations. They also aim to investigate the use of meta-learning algorithms that incorporate more diverse and structured meta-training tasks for improved performance.

Summary updated

BibTeX
@inproceedings{singh2024learning,
  author = {Avyav Singh and Ekaterina Shutova and Helen Yannakoudakis},
  title = {{Learning New Tasks from a Few Examples with Soft-Label Prototypes}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure