Dynosaur: A Dynamic Growth Paradigm for Instruction-Tuning Data Curation

Da Yin, Xiao Liu, Fan Yin, Ming Zhong, Hritik Bansal, Jiawei Han, Kai-Wei Chang

Introduction

To make Large Language Models (LLMs) helpful and followable, researchers rely on "instruction tuning"—the process of fine-tuning models on (instruction, input, output) triplets. Historically, this data has been sourced through two primary, yet flawed, methods: manual human annotation, which is prohibitively expensive and slow, or "distillation," where smaller models mimic outputs from larger ones like GPT-4. While distillation is cheaper, it often results in models that lack factual grounding and complex problem-solving capabilities.

DYNOSAUR introduces a new paradigm by leveraging the thousands of high-quality, human-annotated NLP datasets already available on platforms like Hugging Face. By automatically transforming these existing datasets into instruction-tuning samples using metadata, DYNOSAUR provides a scalable, grounded, and extremely cost-effective alternative to traditional curation methods.

The DYNOSAUR Pipeline

The core of DYNOSAUR is an automated pipeline that "recycles" existing data into instruction-following formats. This process is driven by metadata rather than the raw data itself, allowing for high efficiency.

The DYNOSAUR pipeline: From metadata collection to instruction data generation and filtering.

1. Metadata Collection

The system first scrapes metadata from the Hugging Face Datasets Platform, including dataset names, descriptions, and specific data fields (e.g., "title," "text," "author").

2. Instruction and Field Generation

Using an LLM (GPT-3.5-turbo), DYNOSAUR generates up to three distinct tasks for each dataset. The LLM identifies which existing fields should serve as the "input" and which should be the "output." This is done through two approaches:

  • Description-Aware: Tasks are aligned with the original creator's intent using the dataset summary.
  • Description-Unaware: The LLM creates novel tasks based solely on field names, often discovering creative uses for the data.

3. Filtering and Organization

A post-processing step removes invalid tasks, such as those referencing non-existent fields or having overlapping input/output requirements. Finally, the raw data from the original datasets is "poured" into these generated templates to create the final instruction-tuning samples.

Experimental Setup and Results

The authors generated DYNOSAUR-full, a dataset containing 801,900 instances across 5,740 valid tasks derived from 2,911 English datasets. The framework was evaluated using T5-3B and LLaMA-7B models across several major benchmarks, including SUPER-NI (1,600+ NLP tasks), LONGFORM (long-text generation), and USER-INSTRUCTION-252 (real-world user prompts).

Key Findings

  • Extreme Cost-Efficiency: DYNOSAUR produced 800,000 samples for approximately **$11.50**. In contrast, the 52,000 samples in the Alpaca dataset cost roughly $500 to generate.
  • Superior Performance: Models fine-tuned on DYNOSAUR consistently outperformed those trained on Alpaca, Instruction GPT-4, and Dolly. On the SUPER-NI benchmark, the T5-3B model saw a 2.5 to 22 point ROUGE-L improvement over its competitors.
  • Higher Data Quality: Human evaluation revealed that 84% of DYNOSAUR’s generated instances were completely correct, significantly higher than the 54% reported for the "Self-Instruct" method.
  • Groundedness: The results suggest that instructions grounded in real-world, human-verified data are more effective for model learning than "hallucinated" data where an LLM generates both the prompt and the answer.

Continual Learning and Growth

Beyond static dataset creation, the authors explored how models can grow as new datasets are added to repositories. They found that an "Instruction-Diverse" Replay Strategy—replaying old tasks that are semantically most different from new ones—is the most effective way to prevent "catastrophic forgetting" while improving generalization to unseen prompts.

Conclusion and Future Work

DYNOSAUR demonstrates that metadata is a powerful lever for data curation. By showing that LLMs can design complex, grounded tasks from simple field names and summaries, the framework opens the door for massive-scale, low-cost model training. While currently limited to English and containing a minor noise rate (16%), future iterations aim to expand into multilingual datasets and scale the number of instances per task to further enhance model performance.

Summary updated

BibTeX
@inproceedings{yin2023dynosaur,
  author = {Da Yin and Xiao Liu and Fan Yin and Ming Zhong and Hritik Bansal and Jiawei Han and Kai-Wei Chang},
  title = {{Dynosaur: A Dynamic Growth Paradigm for Instruction-Tuning Data Curation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  doi = {10.18653/v1/2023.emnlp-main.245},
}
Figure