Few-Shot Text Generation with Natural Language Instructions

Timo Schick, Hinrich Schütze

Timo Schick, Hinrich Schütze
Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing

Abstract

Large Language Models (LLMs) often require massive amounts of labeled data to excel at specific generative tasks. In data-scarce environments, standard fine-tuning frequently leads to overfitting, where models "parrot" training examples rather than learning the underlying task. This paper introduces Generative Pattern-Exploiting Training (GENPET), a method that leverages natural language instructions to enable high-performance text generation with as few as 10 labeled examples.

The Challenge: Bridging Zero-Shot and Fine-Tuning

While LLMs show promise in zero-shot settings (performing tasks based solely on instructions), their performance is often inconsistent. Conversely, standard fine-tuning on small datasets (10–100 examples) typically fails because the model lacks enough signal to generalize, often resulting in verbatim copying of the training set. GENPET addresses this by combining the descriptive power of natural language prompts with robust fine-tuning techniques designed for low-data regimes.

Methodology: The GENPET Framework

GENPET adapts the Pattern-Exploiting Training (PET) framework—originally successful in classification—to generative tasks like summarization and headline writing. The framework relies on several key innovations:

1. The Power of Decoder Prefixes

A critical finding of the research is that instructions are significantly more effective when processed by the decoder of an encoder-decoder model (like PEGASUS) rather than the encoder. By providing the instruction as a prefix to the generated text (e.g., starting the decoder with "Short Summary: "), the model is more effectively "primed" to produce the desired output format.

Comparison of encoder-side vs. decoder-side instruction processing.

2. Instruction Ensembles and Distillation

Because the phrasing of a prompt (e.g., "TL;DR:" vs. "Summary:") can drastically alter model performance, GENPET utilizes multiple instructions for each task. The model generates candidate outputs for each prompt, and a "knowledge distillation" process is used to train a final, single model that consolidates the strengths of all individual prompts.

Task-specific decoder prefixes used in GENPET across various benchmarks.

3. Anti-Overfitting via Unsupervised Scoring

To prevent the model from simply memorizing training data, the authors implement an unsupervised scoring mechanism. They use an untrained base model to evaluate generated text. If a candidate output is highly probable only to the fine-tuned model but not to the base model, it is flagged as potential "parroting" and discarded. This ensures the model learns to generate natural, context-appropriate text rather than just repeating training phrases.

Experimental Evaluation

The researchers evaluated GENPET across six diverse English benchmarks, including AESLC (email subjects), Gigaword (headlines), and CNN/DailyMail (bulleted highlights).

Key Results

  • 10-Shot Superiority: In the ultra-lean 10-example setting, GENPET achieved an average ROUGE-1 improvement of 7.20 points over standard fine-tuning (31.63 vs 24.43).
  • Qualitative Gains: Unlike baseline models that often "hallucinated" facts or copied text verbatim, GENPET produced fluent, factual summaries that remained faithful to the source material.
  • Robustness: By training on all instructions simultaneously (joint training), the model acted as its own regularizer, further preventing the overfitting common in few-shot learning.

Conclusion

GENPET demonstrates that natural language instructions are a powerful tool for data-efficient fine-tuning. By moving instructions to the decoder and using untrained models as "BS detectors," GENPET allows language models to achieve high-quality generation with a fraction of the data typically required. Future work remains to automate prompt design and expand these techniques to non-English languages.

Summary updated

BibTeX
@inproceedings{schick2021few,
  author = {Timo Schick and Hinrich Schütze},
  title = {{Few-Shot Text Generation with Natural Language Instructions}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  url = {https://openreview.net/forum?id=hpsRGuzuq4H},
}
Figure