Abstract
Some NLP tasks can be solved in a fully unsupervised fashion by providing a pretrained language model with "task descriptions" in natural language. While this approach underperforms its supervised counterpart, we show in this work that the two ideas can be combined: We introduce Pattern-Exploiting Training (PET), a semi-supervised training procedure that reformulates input examples as cloze-style phrases to help language models understand a given task. These phrases are then used to assign soft labels to a large set of unlabeled examples. Finally, standard supervised training is performed on the resulting training set. For several tasks and languages, PET outperforms supervised training and strong semi-supervised approaches in low-resource settings by a large margin.
Introduction
In real-world natural language processing (NLP), obtaining large labeled datasets is often impractical. Standard supervised learning typically struggles in few-shot regimes, where only 10 to 100 examples are available, as neural networks fail to grasp the core concept of a task from such limited data. Conversely, zero-shot learning using natural language prompts (e.g., appending "It was ___" to a review) leverages the latent knowledge of pretrained language models (PLMs) but often fails to match the performance of supervised methods when even a small amount of data is present.
This paper introduces Pattern-Exploiting Training (PET), a method that bridges this gap. By combining the intuitive power of natural language task descriptions with an ensemble-based knowledge distillation process, PET enables PLMs to learn effectively from tiny datasets, functioning as a powerful prior that makes a handful of examples as effective as thousands.
Methodology
Pattern-Exploiting Training (PET)
PET operates through a multi-step pipeline designed to extract and amplify the task-specific knowledge within a PLM:
- Pattern-Verbalizer Pairs (PVPs): Inputs are converted into cloze questions using manually defined patterns. For example, a sentiment task might use the pattern
text + " It was [MASK].". A "verbalizer" then maps labels to vocabulary words (e.g., Positive to "great", Negative to "bad"). - Ensemble Finetuning: A separate PLM is finetuned for each PVP on the small labeled dataset. To prevent catastrophic forgetting, an auxiliary masked language modeling loss is used on unlabeled data during this phase.
- Soft-Labeling: The ensemble of finetuned models predicts soft labels (probability distributions) for a large set of unlabeled examples.
- Knowledge Distillation: A final standard sequence classifier is trained on this large, soft-labeled dataset, distilling the ensemble's collective knowledge into a single efficient model.
Iterative PET (iPET)
To mitigate the risk of poor manual patterns, the authors propose iPET. This iterative variant trains multiple generations of models. Each new generation is trained on data labeled by a random subset of the previous generation, allowing models to learn from one another and ensuring that weak patterns do not dominate the final training set.
Experimental Results
The authors evaluated PET and iPET across several benchmarks, including Yelp Reviews, AG’s News, Yahoo Questions, and MNLI, artificially restricting training sets to as few as 10 examples.
As shown in Table 1, PET and iPET consistently outperform state-of-the-art semi-supervised baselines like UDA and MixText. In the most extreme low-resource setting (|T|=10), PET-based methods provide massive gains. For instance, on the AG's News topic classification task, iPET achieves 87.5% accuracy, significantly higher than MixText's 81.1%.
On the Yelp sentiment analysis task with only 10 examples, standard supervised learning often performs near random chance (~20%), while PET-based approaches achieve over 50% accuracy. The method also demonstrated robust cross-lingual transfer capabilities on the x-stance dataset, improving performance in zero-shot settings for languages like Italian when trained on German or French data.
Discussion and Conclusion
The success of PET highlights that framing tasks as natural language questions acts as a powerful inductive bias for PLMs. By using ensembles, the method effectively neutralizes the sensitivity of models to specific prompt wording. While PET currently relies on human intuition for pattern design, it represents a significant step forward in few-shot learning. Future work may focus on automating the discovery of optimal patterns and verbalizers to further reduce the need for manual engineering.
Summary updated