Generating Datasets with Pretrained Language Models

Timo Schick, Hinrich Schütze

Abstract: Timo Schick, Hinrich Schütze. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing.

2021.

Introduction

To enable computers to understand the semantic meaning of sentences—a task known as sentence embedding—researchers typically rely on two paths: massive human-labeled datasets, which are expensive and slow to produce, or unsupervised methods, which often yield lower-quality results. While large pretrained language models (PLMs) like GPT-3 possess immense linguistic knowledge, they are often too computationally expensive for real-time sentence comparison. Conversely, smaller, faster models like BERT or RoBERTa require specific fine-tuning to produce high-quality embeddings.

This paper introduces DINO (Datasets from Instructions), a framework that addresses this "data bottleneck" by using large PLMs to automatically generate high-quality labeled datasets. By prompting a large model to "hallucinate" sentence pairs with specific similarity levels, the authors can train smaller, more efficient models that outperform traditional unsupervised baselines and even rival supervised ones.

The DINO Framework

Instead of employing human annotators, DINO provides "instructions" to a large PLM (such as GPT2-XL) to generate training data. The process involves several key components:

  1. Instructional Prompting: The model is given a template task, such as: "Task: Write two sentences that [mean the same thing / are somewhat similar / are on different topics]. Sentence 1: '[Input]'. Sentence 2:"
  2. Generation Modes: The authors explore two variants: STS-Dino-$x_2$, where the model generates a matching sentence for an existing input, and STS-Dino-$x_1x_2$, where the model generates both sentences from scratch, requiring no external data.
  3. Self-Debiasing: To prevent the model from generating trivial or nearly identical sentences for "somewhat similar" tasks, a self-debiasing algorithm is used. This penalizes the model if the words chosen for a lower-similarity pair are too likely to appear in a high-similarity context.
  4. Distillation: The resulting synthetic dataset is used to fine-tune a standard Sentence-RoBERTa model.
Examples of sentence pairs generated by DINO across different similarity levels (y), with human verification marks indicating accuracy.

Key Insights and Results

Large Models as Synthetic Crowdworkers

The research demonstrates that large PLMs have internalized enough linguistic knowledge to act as effective annotators. While human evaluation showed that GPT2-XL only correctly labeled "same meaning" pairs about 47% of the time, the downstream models trained on this data performed exceptionally well. This suggests that deep learning models are remarkably resilient to noise in synthetic data, provided the overall signal remains strong.

Performance Benchmarks

DINO significantly outperformed all unsupervised baselines, including GloVe averages and BERT CLS tokens. More impressively, Sentence-RoBERTa trained on DINO's synthetic data outperformed versions of S-BERT and S-RoBERTa trained on actual human-labeled Natural Language Inference (NLI) data across four out of six Semantic Textual Similarity (STS) datasets.

Performance comparison of DINO against supervised (sup.) and unsupervised (unsup.) baselines across various STS benchmarks.

Conclusion and Future Work

DINO proves that synthetic data generation is a viable alternative to manual annotation for sentence embeddings. While the model occasionally struggled with generating diverse topics or perfect paraphrases, the efficiency of the final distilled model makes it highly suitable for production environments. Future work may involve using even larger models (like GPT-3) as generators or implementing automated filtering steps to further refine the quality of the synthetic datasets.

Summary updated

BibTeX
@inproceedings{schick2021generating,
  author = {Timo Schick and Hinrich Schütze},
  title = {{Generating Datasets with Pretrained Language Models}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  url = {https://openreview.net/forum?id=hpsRGuzuq4H},
}
Figure