Timo Schick, Hinrich Schütze
Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing
Abstract
Large Language Models (LLMs) often require massive amounts of labeled data to excel at specific generative tasks. In data-scarce environments, standard fine-tuning frequently leads to overfitting, where models "parrot" training examples rather than learning the underlying task. This paper introduces Generative Pattern-Exploiting Training (GENPET), a method that leverages natural language instructions to enable high-performance text generation with as few as 10 labeled examples.
The Challenge: Bridging Zero-Shot and Fine-Tuning
While LLMs show promise in zero-shot settings (performing tasks based solely on instructions), their performance is often inconsistent. Conversely, standard fine-tuning on small datasets (10–100 examples) typically fails because the model lacks enough signal to generalize, often resulting in verbatim copying of the training set. GENPET addresses this by combining the descriptive power of natural language prompts with robust fine-tuning techniques designed for low-data regimes.
Methodology: The GENPET Framework
GENPET adapts the Pattern-Exploiting Training (PET) framework—originally successful in classification—to generative tasks like summarization and headline writing. The framework relies on several key innovations:
1. The Power of Decoder Prefixes
A critical finding of the research is that instructions are significantly more effective when processed by the decoder of an encoder-decoder model (like PEGASUS) rather than the encoder. By providing the instruction as a prefix to the generated text (e.g., starting the decoder with "Short Summary: "), the model is more effectively "primed" to produce the desired output format.
2. Instruction Ensembles and Distillation
Because the phrasing of a prompt (e.g., "TL;DR:" vs. "Summary:") can drastically alter model performance, GENPET utilizes multiple instructions for each task. The model generates candidate outputs for each prompt, and a "knowledge distillation" process is used to train a final, single model that consolidates the strengths of all individual prompts.
3. Anti-Overfitting via Unsupervised Scoring
To prevent the model from simply memorizing training data, the authors implement an unsupervised scoring mechanism. They use an untrained base model to evaluate generated text. If a candidate output is highly probable only to the fine-tuned model but not to the base model, it is flagged as potential "parroting" and discarded. This ensures the model learns to generate natural, context-appropriate text rather than just repeating training phrases.
Experimental Evaluation
The researchers evaluated GENPET across six diverse English benchmarks, including AESLC (email subjects), Gigaword (headlines), and CNN/DailyMail (bulleted highlights).
Key Results
- 10-Shot Superiority: In the ultra-lean 10-example setting, GENPET achieved an average ROUGE-1 improvement of 7.20 points over standard fine-tuning (31.63 vs 24.43).
- Qualitative Gains: Unlike baseline models that often "hallucinated" facts or copied text verbatim, GENPET produced fluent, factual summaries that remained faithful to the source material.
- Robustness: By training on all instructions simultaneously (joint training), the model acted as its own regularizer, further preventing the overfitting common in few-shot learning.
Conclusion
GENPET demonstrates that natural language instructions are a powerful tool for data-efficient fine-tuning. By moving instructions to the decoder and using untrained models as "BS detectors," GENPET allows language models to achieve high-quality generation with a fraction of the data typically required. Future work remains to automate prompt design and expand these techniques to non-English languages.
Summary updated