AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback

Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, Tatsunori B. Hashimoto

TL;DR

AlpacaFarm is an open-source simulation sandbox that mimics human feedback using LLMs, reducing the cost of alignment research by 50x. It provides validated evaluation protocols and reference implementations (PPO, DPO, etc.) that closely replicate real-world human training dynamics, including complex phenomena like reward model over-optimization.

What Problem Does This Paper Address?

While Reinforcement Learning from Human Feedback (RLHF) is central to making models like ChatGPT follow instructions, it remains poorly understood and notoriously difficult to research. The authors identify three major bottlenecks:

  1. High Cost: Gathering human pairwise preferences is prohibitively expensive for typical research labs (costing thousands of dollars per experiment).
  2. Evaluation Gap: Open-ended instruction-following lacks trustworthy, automated, and reproducible evaluation metrics.
  3. Missing Baselines: The community lacks validated, open-source reference implementations of standard alignment algorithms.

AlpacaFarm solves this by replacing expensive human annotators with a low-cost, highly accurate LLM-based simulation environment.

Figure 1: AlpacaFarm is a simulation sandbox that enables fast and cheap experimentation on LLMs that learn from human feedback. It simulates human feedback with (oracle) API LLMs, provides a validated evaluation protocol, and offers a suite of reference method implementations. Researchers can rapidly iterate on model development and transfer their methods to training on human data to maximize performance.

Methods

AlpacaFarm simulates the "Learning from Pairwise Feedback" (LPF) workflow. Instead of querying humans to choose the better of two model responses, it queries an ensemble of API-based LLMs (such as GPT-4, ChatGPT, and Davinci003).

To make the simulation realistic, the authors didn't just use a single deterministic prompt. Instead, they engineered a pool of 13 distinct simulated annotators varying in prompt format, in-context examples, and underlying models. Crucially, they introduced 25% label-flipping noise to emulate the natural intra-annotator variability seen in real human crowdsourcing.

The framework includes reference implementations for several major alignment algorithms, categorized into:

  • Direct Pairwise Learners: Binary FeedME, Binary Reward Conditioning, and Direct Preference Optimization (DPO).
  • Surrogate Reward Maximizers: PPO, Quark, Best-of-n sampling, and Expert Iteration.

Datasets

The authors repurposed the original 52k Alpaca dataset into four distinct splits:

  • Supervised Fine-Tuning (SFT) split: 10k instructions to train the base model.
  • Pairwise Preference (PREF) split: 10k instructions used to collect simulated/human feedback.
  • Unlabeled split: 20k instructions for unlabeled RL exploration (e.g., PPO rollouts).
  • Validation split: 2k instructions for hyperparameter tuning.

For evaluation, they constructed a diverse 805-instruction evaluation set by combining several open-source datasets (Self-Instruct, OASST, Anthropic Helpful, Vicuna, and Koala) to mimic real-world user interaction distributions.

Key Insights and Ideas

  • Noise is a Feature, Not a Bug: The most surprising "aha moment" in the paper is that perfect LLM annotators fail to replicate human training dynamics. If you use a clean, single-prompt GPT-4 annotator, algorithms can optimize against it indefinitely without degrading. However, when you inject human-like noise and variability, the simulation successfully replicates reward model over-optimization (where a model learns to exploit an imperfect reward model, causing actual response quality to drop).
  • High Correlation at Low Cost: The simulated annotators achieve a 65% agreement rate with a human majority vote—essentially identical to the 66% agreement rate between different human annotators—but at 50x lower cost ($6 vs $300 per 1,000 examples).
Simulated annotator agreement with human majority vote versus cost per 1000 examples.

Key Results

  • End-to-End Validation: The authors trained 11 different models using both simulated feedback and real human feedback. The resulting model rankings between the simulation and human evaluation showed a staggering Spearman correlation of 0.98.
  • PPO Dominates: Among the tested methods, PPO with a surrogate reward model performed the best, boosting the win-rate of a LLaMA-7B model against Davinci003 from 44% to 55% under human evaluation.
  • The Length Bias: Analysis of the outputs revealed that RLHF and other preference-optimization methods heavily shift the model toward generating longer responses and lists, a stylistic preference shared by both human and LLM annotators.
High correlation between model win-rates evaluated in simulation vs. evaluated by humans.

Limitations and Future Work

  • Simplicity: The evaluation and training instructions are currently limited to simple, single-turn prompts. Real-world deployment often requires multi-turn dialogue capabilities.
  • Hyperparameter Scaling: While model rankings transfer perfectly from simulation to the real world, the exact hyperparameter scales (like the optimal KL-regularization coefficient in PPO) differ due to differences in the reward model's absolute scale.
  • Base Model Constraints: All experiments were anchored on a LLaMA-7B base model; behavior might shift with larger or different base architectures.

Summary updated

Abstract

Large language models (LLMs) such as ChatGPT have seen widespread adoption due to their strong instruction-following abilities. Developing these LLMs involves a complex yet poorly understood workflow requiring training with human feedback. Replicating and understanding this instruction-following requires tackling three major challenges: the high cost of data collection, the lack of trustworthy evaluation, and the absence of reference method implementations. We address these challenges with AlpacaFarm, a simulator that enables research and development for learning from feedback at a low cost. First, we design LLM prompts to simulate human feedback that are 50x cheaper than crowdworkers and display high agreement with humans. Second, we propose an automatic evaluation and validate it against human instructions obtained on real-world interactions. Third, we contribute reference implementations for several methods (PPO, DPO, best-of-n, expert iteration, and more) that learn from pairwise feedback. Finally, as an end-to-end validation of AlpacaFarm, we train and evaluate eleven models on 10k pairs of real human feedback and show that rankings of models trained in AlpacaFarm match rankings of models trained on human data. As a demonstration of the research possible in AlpacaFarm, we find that methods that use a reward model can substantially improve over supervised fine-tuning and that our reference PPO implementation leads to a +10% improvement in win-rate against Davinci003. We release all components of AlpacaFarm at https://github.com/tatsu-lab/alpaca_farm.
BibTeX
@inproceedings{dubois2023alpacafarm,
  author = {Yann Dubois and Xuechen Li and Rohan Taori and Tianyi Zhang and Ishaan Gulrajani and Jimmy Ba and Carlos Guestrin and Percy Liang and Tatsunori B. Hashimoto},
  title = {{AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2023},
  url = {https://openreview.net/forum?id=4hturzLcKX},
}
Figure