AlpaGasus: Training A Better Alpaca with Fewer Data

Lichang Chen, Shiyang Li, Jun Yan, Hai Wang, Kalpa Gunaratna, Vikas Yadav, Zheng Tang, Vijay Srinivasan, Tianyi Zhou, Heng Huang, Hongxia Jin

TL;DR

AlpaGasus demonstrates that for instruction fine-tuning (IFT), a small amount of high-quality data is significantly better than a large amount of noisy data. By using ChatGPT to filter the 52k Alpaca dataset down to just 9k "gold" samples, the authors trained a model that outperforms the original while reducing training time by 5.7x.

What Problem Does This Paper Address?

Instruction Fine-Tuning (IFT) is the process that transforms a raw base model (like LLaMA) into a helpful assistant (like Alpaca or ChatGPT). Most open-source IFT models rely on datasets generated by "teacher" models. For instance, the original Alpaca model was trained on 52,000 instruction-response pairs generated by OpenAI’s text-davinci-003.

However, these machine-generated datasets are surprisingly noisy. They often contain incorrect, irrelevant, or nonsensical responses that can mislead the model during training. While "cleaning" these datasets manually would be ideal, human annotation is prohibitively expensive and slow. The authors of AlpaGasus set out to automate this data-centric cleaning process to see if "less is more" in the world of LLM alignment.

Methods

The authors proposed a simple, automated data-filtering pipeline using a strong LLM (ChatGPT) as an "auto-grader."

  1. Scoring: They prompted ChatGPT to evaluate each (instruction, input, response) triplet from the Alpaca dataset on a scale of 0 to 5, focusing on the dimension of "accuracy."
  2. Filtering: They analyzed the score distribution (seen in Figure 4 below) and chose a threshold of $\tau = 4.5$. This filtered the 52,000 samples down to a high-quality subset of roughly 9,000 samples.
  3. Training: They fine-tuned LLaMA-7B and 13B models on this 9k subset using the exact same hyperparameters and training script as the original Alpaca.
Figure 4: Histogram of Scores (Al- paca Dataset).

The overall pipeline is illustrated here: Figure 2: The AlpaGasus fine-tuning pipeline, showing how low-quality triplets are identified by the ChatGPT-based evaluator and discarded before training.

Figure 2: The fine-tuning pipeline of ALPAGASUS. We prompt ChatGPT as our auto-grader to score each training triplet on a scale of 0 to 5. We then use the exact same instruction fine-tuning script of ALPACA to train ALPAGASUS on the filtered data with scores higher than a threshold.

Datasets

  • Training Data: The primary source was the Alpaca 52k dataset. The authors also applied their method to the human-written Databricks-dolly-15k and the GPT4LLM dataset to prove versatility.
  • Evaluation Sets: To avoid bias, the models were tested against four diverse instruction sets: Vicuna (80 samples), Koala (180 samples), WizardLM (218 samples), and Self-Instruct (252 samples).
  • Benchmarks: Standard benchmarks like MMLU (knowledge), BBH (reasoning), DROP (reading comprehension), and HumanEval (coding) were used for quantitative comparison.

Key Insights and Ideas

  • The "Noise" is Detrimental: The fact that 9k samples outperform 52k samples implies that the remaining 43k samples weren't just redundant—they were actually harmful to the model's alignment.
  • LLM-as-a-Judge Works for Data Selection: Stronger models like ChatGPT are capable of identifying subtle errors in the outputs of slightly weaker models (like text-davinci-003), making them effective automated filters.
  • Data-Centric IFT: This shifts the focus from scaling up data volume to "prudent management" of data quality. This paradigm is much more sustainable for researchers with limited compute.

Key Results

  • Superior Performance: AlpaGasus-9k significantly outperformed Alpaca-52k across all four test sets. As shown in Figure 5, the 13B variant of AlpaGasus wins or ties against the original Alpaca in the vast majority of cases.
  • Efficiency Gains: Training time for the 7B model dropped from 80 minutes to just 14 minutes on 4x NVIDIA A100 GPUs.
  • Teacher Matching: AlpaGasus-13B matched over 90% of the performance of its teacher model (text-davinci-003) on test tasks, despite being trained on a fraction of the data.
  • Generalization: The filtering strategy also improved models trained on human-written data (Dolly), showing that even "seasoned writers" produce noise that LLMs can help filter.
Figure 5: Main results comparing AlpaGasus and Alpaca. AlpaGasus-9k achieves much better performance than Alpaca-52k on all four test sets.

Limitations and Future Work

  • Model Scale: The study focused on 7B and 13B models. The authors plan to extend this to larger models like 33B, 65B, or 175B to see if the "quality > quantity" rule holds at the extreme scale.
  • Category Imbalance: The authors noted that their automated filtering led to a very high rejection rate for coding-related data (88% filtered out). This resulted in AlpaGasus having slightly weaker coding skills compared to other categories. Future work could involve "balanced filtering" to ensure diverse skill sets are maintained.
  • Rating Scales: The current 5-point scale might be too coarse. Exploring finer-grained scoring systems could lead to even better data selection.

Summary updated

Abstract

Large language models (LLMs) strengthen instruction-following capability through instruction-finetuning (IFT) on supervised instruction/response data. However, widely used IFT datasets (e.g., Alpaca's 52k data) surprisingly contain many low-quality instances with incorrect or irrelevant responses, which are misleading and detrimental to IFT. In this paper, we propose a simple and effective data selection strategy that automatically identifies and filters out low-quality data using a strong LLM (e.g., ChatGPT). To this end, we introduce AlpaGasus, which is finetuned on only 9k high-quality data filtered from the 52k Alpaca data. AlpaGasus significantly outperforms the original Alpaca as evaluated by GPT-4 on multiple test sets and the controlled human evaluation. Its 13B variant matches $>90\%$ performance of its teacher LLM (i.e., Text-Davinci-003 generating the 52k data) on test tasks. It also provides 5.7x faster training, reducing the training time for a 7B variant from 80 minutes (for Alpaca) to 14 minutes. Moreover, the experiments prove the efficacy of our method across diverse datasets, base models, and LLM filters. Overall, AlpaGasus demonstrates a novel data-centric IFT paradigm that can be generally applied to instruction-tuning data, leading to faster training and better instruction-following models. Our project page is available at: https://lichang-chen.github.io/AlpaGasus/
BibTeX
@inproceedings{chen2024alpagasus,
  author = {Lichang Chen and Shiyang Li and Jun Yan and Hai Wang and Kalpa Gunaratna and Vikas Yadav and Zheng Tang and Vijay Srinivasan and Tianyi Zhou and Heng Huang and Hongxia Jin},
  title = {{AlpaGasus: Training A Better Alpaca with Fewer Data}},
  booktitle = {International Conference on Learning Representations},
  year = {2024},
  url = {https://openreview.net/forum?id=FdVXgSJhvz},
}
Figure