Abstract
Researchers have demonstrated that "small" language models (like ALBERT) can match or even outperform massive models like GPT-3 on few-shot learning tasks. By using Pattern-Exploiting Training (PET)—a method that reformulates tasks as fill-in-the-blank questions—a model with 223 million parameters can beat one with 175 billion parameters while being significantly "greener" and more accessible.
The Scaling Dilemma in NLP
The recent trend in Natural Language Processing (NLP) has been "bigger is better." Models like GPT-3 achieve incredible results in few-shot settings—learning from just a handful of examples—by using priming. In priming, the model is given a few examples in its context window and asked to complete the next one without any parameter updates.
However, this "Goliath" approach has three major drawbacks:
- Accessibility & Cost: These models require massive compute clusters, making them unusable for most researchers.
- Environmental Impact: Training and running these models leaves a massive carbon footprint.
- Scaling Limits: Priming doesn't scale well. Because language models have a limited context window, you can only feed them a few examples before running out of space.
This paper investigates whether GPT-3 level few-shot performance can be achieved using models that are three orders of magnitude smaller.
Methodology: Pattern-Exploiting Training (PET)
The authors utilize and extend Pattern-Exploiting Training (PET). Instead of traditional fine-tuning, which often fails with very little data, PET turns a task into a "cloze" (fill-in-the-blank) question using Pattern-Verbalizer Pairs (PVPs).
The PET Recipe:
- PVP Creation: A template (pattern) and a mapping (verbalizer) are created. For example, for sentiment analysis, the input "The food was great" becomes "The food was great. It was [MASK]." The verbalizer maps the label "Positive" to the word "great."
- Ensemble Training: Several models are fine-tuned on different patterns using a small labeled dataset (e.g., 32 examples).
- Distillation: These models label a large set of unlabeled data. A final, standard classifier is then trained on this newly labeled "soft" dataset for easy deployment.
- Iterative PET (iPET): An iterative version where models of increasing quality label data for each other across multiple generations.
A key novelty in this work is the handling of multiple masks, allowing the model to predict multi-token answers (like "terrible" being split into "terri" and "ble"), which is essential for complex reasoning tasks.
Experimental Setup
The researchers evaluated their approach on SuperGLUE, a notoriously difficult benchmark for natural language understanding. They introduced FewGLUE, a subset consisting of only 32 labeled examples per task, plus a larger set of unlabeled data to simulate a real-world few-shot scenario.
Results: David vs. Goliath
The results represent a significant victory for model efficiency. On the SuperGLUE benchmark, ALBERT with iPET achieved a score of 76.8, significantly higher than GPT-3’s 73.2, despite being 785 times smaller.
Key Insights:
- Bidirectionality: PET works exceptionally well with ALBERT because it is "bidirectional" (looking at words to the left and right of the mask), whereas GPT-3 is "unidirectional," making it less efficient at filling in blanks in the middle of a sentence.
- Efficiency: ALBERT-xxlarge has only 0.1% of the parameters of GPT-3 and can be trained in a few hours on a single consumer GPU.
- Consistency: PET outperformed GPT-3 across almost all individual tasks in the SuperGLUE suite and continued to improve as more data was provided, unlike GPT-3 which plateaus due to context window limits.
Conclusion and Future Work
This research proves that massive scale is not the only path to few-shot learning. By reformulating tasks to exploit the pre-existing knowledge in smaller models, we can achieve state-of-the-art results more sustainably. Future work includes automating pattern discovery and adapting PET for generative tasks like summarization or translation.
Summary updated