Generating Datasets with Pretrained Language Models

Timo Schick, Hinrich Schütze

TL;DR

Researchers have found a way to bypass expensive human labeling by using Large Language Models (LLMs) to "hallucinate" entire labeled datasets for training. By giving a model like GPT-2 XL specific instructions and a clever "self-debiasing" trick, they generated a massive dataset called STS-Dino that allows smaller models to achieve state-of-the-art performance in sentence embeddings—sometimes even outperforming models trained on human-annotated data.

What Problem Does This Paper Address?

To make sense of text, computers need "sentence embeddings"—numerical representations that capture the meaning of a sentence. While models like BERT are great at understanding words, they are surprisingly bad at sentence-level meaning right out of the box.

The gold standard for fixing this is fine-tuning them on massive datasets of sentence pairs labeled by humans (e.g., "These two sentences mean the same thing"). However, hiring humans to write and label hundreds of thousands of sentence pairs is slow, expensive, and difficult to scale. The authors ask: Can we use the generative power of massive language models to create these training datasets automatically?

Methods

The authors introduce DINO (Datasets from INstructions), a framework that treats a Pretrained Language Model (PLM) as a synthetic crowdworker.

  1. Instructions: They prompt GPT-2 XL with a task: "Write two sentences that [mean the same thing / are somewhat similar / are on completely different topics]."
  2. Two Generation Modes:
    • $x_2$ only: The model is given an existing sentence and asked to generate a partner.
  3. $x_1$ and $x_2$: The model generates both sentences from scratch, requiring zero external data.
  4. Self-Debiasing: This is the "secret sauce." To ensure a "somewhat similar" sentence doesn't accidentally become "exactly the same," the authors use a technique that penalizes the model if it generates words that would be highly likely for a different label. It forces the model to stay within the specific "semantic lane" requested by the instruction.
  5. Distillation: They use these millions of synthetic pairs to train a much smaller, more efficient model (Sentence-RoBERTa).

Datasets

The authors produced and released STS-Dino (STS-🦖), the first textual similarity dataset generated completely automatically.

  • Training/Evaluation: They evaluated their method on standard benchmarks: STS 2012–2016, the STS benchmark (STSb), and SICK-Relatedness.
  • Comparison: They compared DINO-trained models against "unsupervised" baselines (like GloVe or raw BERT) and "supervised" models trained on human-labeled NLI (Natural Language Inference) data.

Key Insights and Ideas

  • LLMs as Data Factories: Large models are often too bulky to use for everyday tasks. DINO shows we can use a "giant" model once to create a dataset, then throw the giant away and use the dataset to train a "small" model that is fast and accurate.
  • The Power of "Not": The self-debiasing results show that telling a model what not to be is just as important as telling it what to be. By penalizing "perfect matches" when asking for "partial matches," the data quality improved significantly.
  • Synthetic > Human? In a surprising twist, the fully unsupervised DINO approach outperformed models trained on human-annotated NLI data on four out of six STS tasks. This suggests that the scale and task-specificity of synthetic data can sometimes outweigh the precision of human labels.

Key Results

  • Performance: S-RoBERTa trained on DINO achieved an average Spearman correlation of 75.20 across STS tasks, beating the best previous unsupervised baseline (69.57).
  • Efficiency: Generating the datasets took about 48 hours on two consumer-grade GPUs, while training the final "student" model took less than 2 hours.
  • Label Smoothing: Because synthetic data is "noisy" (the model sometimes makes mistakes), the authors found that using "label smoothing"—treating a 1.0 similarity score as a 0.9—helped the model learn better by not over-trusting the synthetic labels.

Limitations and Future Work

  • The "Copy-Paste" Problem: GPT-2 XL sometimes struggled to generate truly diverse paraphrases. For the "mean the same thing" task, it often just repeated the sentence or changed a single word.
  • Semantic Nuance: In the "different topics" task, the model sometimes just flipped a word to its antonym (e.g., "closed" to "open"). While these are technically different, they are semantically very related, which can confuse the training process.
  • Future Directions: The authors suggest exploring more complex instructions or adding a "filtering" step where a second model checks the quality of the generated pairs before they are used for training.

Summary updated

Abstract

To obtain high-quality sentence embeddings from pretrained language models (PLMs), they must either be augmented with additional pretraining objectives or finetuned on a large set of labeled text pairs. While the latter approach typically outperforms the former, it requires great human effort to generate suitable datasets of sufficient size. In this paper, we show how PLMs can be leveraged to obtain high-quality sentence embeddings without the need for labeled data, finetuning or modifications to the pretraining objective: We utilize the generative abilities of large and high-performing PLMs to generate entire datasets of labeled text pairs from scratch, which we then use for finetuning much smaller and more efficient models. Our fully unsupervised approach outperforms strong baselines on several semantic textual similarity datasets.
BibTeX
@inproceedings{schick2021generating,
  author = {Timo Schick and Hinrich Schütze},
  title = {{Generating Datasets with Pretrained Language Models}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  url = {https://openreview.net/forum?id=hpsRGuzuq4H},
}
Figure