It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners

Timo Schick, Hinrich Schütze

Abstract

When scaled to hundreds of billions of parameters, pretrained language models such as GPT-3 (Brown et al., 2020) achieve remarkable few-shot performance. However, enormous amounts of compute are required for training and applying such big models, resulting in a large carbon footprint and making it difficult for researchers and practitioners to use them. We show that performance similar to GPT-3 can be obtained with language models that are much "greener" in that their parameter count is several orders of magnitude smaller. This is achieved by converting textual inputs into cloze questions that contain a task description, combined with gradient-based optimization; exploiting unlabeled data gives further improvements. We identify key factors required for successful natural language understanding with small language models.

Introduction

The release of GPT-3 popularized the concept of priming (in-context learning), where a model performs a task based on a few examples provided in its prompt without weight updates. While effective, priming requires massive models to be successful, leading to high computational costs and environmental impact. Furthermore, the fixed context window of these models limits the number of examples that can be utilized. This paper introduces a more efficient alternative: achieving state-of-the-art few-shot performance using "Green AI"—models small enough to run on a single commodity GPU.

Methodology: Pattern-Exploiting Training (PET)

Instead of relying on in-context priming, the authors propose Pattern-Exploiting Training (PET). This method reformulates natural language understanding tasks as cloze (fill-in-the-blank) questions, allowing the model to leverage its pretraining knowledge through gradient-based fine-tuning.

Illustration of Pattern-Exploiting Training (PET) converting an entailment task into a cloze question.

Core Components of PET:

  1. Pattern-Verbalizer Pairs (PVPs): Inputs are wrapped in a pattern that includes a mask. For instance, a sentiment analysis task might append "It was __." to a review. The model predicts a "verbalizer" token (e.g., "great" or "terrible") to determine the label.
  2. Handling Multi-Token Answers: The authors extended PET to handle multi-token predictions using an autoregressive decoding strategy and a multi-class hinge loss.
  3. Knowledge Distillation: To mitigate sensitivity to specific prompt phrasing, an ensemble of models is trained on various patterns. This ensemble generates "soft labels" for unlabeled data, which is then used to train a final, lightweight classifier.
  4. Iterative PET (iPET): An iterative variant where models are trained in generations, with each generation labeling more data for the next based on high-confidence predictions.

Experimental Setup

The approach was evaluated on the SuperGLUE benchmark. To ensure a fair comparison with GPT-3, the authors introduced FewGLUE, a subset of SuperGLUE restricted to 32 labeled examples per task, supplemented by up to 20,000 unlabeled examples for the distillation phase.

Key Results and Efficiency

The results demonstrate that small models can indeed be powerful few-shot learners. Using iPET, an ALBERT-xxlarge-v2 model (223M parameters) achieved an average score of 75.4 on SuperGLUE, outperforming the 175B parameter GPT-3, which scored 71.8.

SuperGLUE performance vs. model parameters, comparing GPT-3 against PET and iPET.

As shown in the figure above, PET and iPET achieve superior performance while being three orders of magnitude smaller (0.1% the size of GPT-3). Training a single PET model takes approximately 45 minutes on a single NVIDIA GTX 1080 Ti, making high-performance NLU accessible to researchers without massive compute clusters.

Discussion

Several key insights emerged from this study:

  • Ensembling Mitigates Prompt Sensitivity: While LMs are sensitive to prompt wording, ensembling multiple patterns stabilizes performance.
  • Bidirectionality is Advantageous: Bidirectional models like ALBERT allow for flexible mask placement, unlike unidirectional models (e.g., GPT-2/3) where the mask must be at the end.
  • Scalability: Unlike priming, which hits a performance ceiling due to context window limits, PET scales effectively as more labeled or unlabeled data becomes available.

Conclusion

This work proves that massive scale is not the only path to few-shot learning. By combining task reformulation with gradient-based optimization and distillation, small models can match or exceed the performance of their giant counterparts. Future research will explore adapting PET for generative tasks and multi-task learning environments.

Summary updated

Abstract

When scaled to hundreds of billions of parameters, pretrained language models such as GPT-3 (Brown et al., 2020) achieve remarkable few-shot performance. However, enormous amounts of compute are required for training and applying such big models, resulting in a large carbon footprint and making it difficult for researchers and practitioners to use them. We show that performance similar to GPT-3 can be obtained with language models that are much "greener" in that their parameter count is several orders of magnitude smaller. This is achieved by converting textual inputs into cloze questions that contain a task description, combined with gradient-based optimization; exploiting unlabeled data gives further improvements. We identify key factors required for successful natural language understanding with small language models.
BibTeX
@inproceedings{schick2021it,
  author = {Timo Schick and Hinrich Schütze},
  title = {{It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
  url = {https://openreview.net/forum?id=sILQotU-uGn},
}
Figure