Learning Preference Model for LLMs via Automatic Preference Data Generation

Shijia Huang, Jianqiao Zhao, Yanyang Li, Liwei WangEMNLP 2023 · OpenReview

Abstract

Despite the advanced capacities of state-of-the-art large language models (LLMs), they suffer from issues of hallucination, stereotype, and misalignment. Preference models (PMs) play an important role in LLM alignment, yet training them predominantly relies on human-annotated data, which limits versatility and scalability. In this paper, we propose AutoPM, a framework for learning preference models via automatic preference data generation. Our approach involves both In-Breadth Data Generation, which elicits pairwise preference data following helpful-honest-harmless (HHH) criteria, and In-Depth Data Generation, which enriches the dataset with responses spanning a wide quality range. Quantitative assessments demonstrate that AutoPM enables LLMs to learn human preferences and align with human values effectively, offering a scalable path to improved LLM performance.

Introduction

Reinforcement Learning from Human Feedback (RLHF) has become the standard for aligning LLMs with human values. This process relies on a Preference Model (PM) that ranks responses to guide the policy model. Traditionally, training these PMs requires thousands of human-labeled comparisons—a process that is slow, expensive, and difficult to scale. While AI Feedback (using models like GPT-4 to rank responses) has been explored, existing methods often lack a systematic framework to ensure data diversity and adherence to safety guidelines. AutoPM addresses this by using a "guideline-driven" approach to automatically generate 42k preference pairs, bypassing the human annotation bottleneck.

Methodology

AutoPM generates synthetic preference data through three distinct strategies designed to capture both broad alignment principles and fine-grained quality nuances.

1. In-Breadth Data Generation

To ensure the model understands the "HHH" criteria, the framework uses pairs of opposite attributes (e.g., "useful" vs. "useless," "fact-based" vs. "fabricated"). An LLM (GPT-3.5) is prompted to generate two responses for the same instruction: one following the positive attribute and one following the negative one. This creates a clear signal for the PM to distinguish between helpful and unhelpful behavior.

2. In-Depth Data Generation

To prevent the PM from becoming a simple binary classifier, AutoPM creates a quality gradient. Starting with a high-quality response, the system applies specific Edit Actions to degrade it:

  • Deletion: Removing useful content.
  • Substitution: Changing content to be inaccurate.
  • Insertion: Adding irrelevant "noise." This forces the PM to learn subtle differences in response quality rather than just surface-level patterns.
Overview of the AutoPM data generation framework, illustrating the In-Breadth and In-Depth strategies.

3. Paired Models Data

Finally, the framework leverages the inherent performance gap between models of different scales. Responses from a strong model (GPT-3.5) serve as "chosen" examples, while responses from a smaller model (OPT-IML 1.3B) serve as "rejected" examples for the same prompt.

Experimental Results

The final preference model was trained by fine-tuning a LLaMA-7B backbone on the 42k-sample synthetic dataset. Evaluation was conducted across several benchmarks, including HHH-Eval, TruthfulQA, and the Vicuna benchmark.

Performance and Consistency

AutoPM-7B achieved an overall accuracy of 0.83 on HHH-Eval, nearly matching the 0.86 score of Anthropic’s 52B parameter model trained on massive human datasets. Crucially, AutoPM’s rankings showed high consistency with both human annotators and GPT-4, outperforming other open-source preference models like PandaLM.

Comparison of AutoPM's evaluation consistency against human annotators and GPT-4 across different model pairings.

Alignment via Rejection Sampling

The study demonstrates that AutoPM can improve existing models without complex Reinforcement Learning (PPO). By generating multiple responses and using AutoPM to select the best one (Rejection Sampling), the truthfulness of models like Alpaca and Vicuna was significantly boosted. For instance, Alpaca-7B’s TruthfulQA score rose from 0.38 to 0.51.

Key Insights

  • Generalization: Even though AutoPM was trained on data from GPT-3.5, it successfully discriminated between responses from stronger models like GPT-4. This suggests the PM learns fundamental principles of preference rather than just mimicking a specific model's style.
  • Negative Constraints: Explicitly prompting the generator to be "useless" or "dishonest" provides a much stronger training signal than random sampling.

Conclusion and Future Work

AutoPM provides a scalable, automated alternative to human-centric alignment. While an expert review found minor noise in the synthetic data (1-2% error rate), the model's performance remains robust. Future research will explore more complex edit actions and the potential for a "self-evolving" alignment loop where models iteratively improve their own preference data.

Summary updated

Abstract

Despite the advanced capacities of the state-of-the-art large language models (LLMs), they suffer from issues of hallucination, stereotype, etc. Preference models play an important role in LLM alignment, yet training preference models predominantly rely on human-annotated data. This reliance limits their versatility and scalability. In this paper, we propose learning the preference model for LLMs via automatic preference data generation (AutoPM). Our approach involves both In-Breadth Data Generation, which elicits pairwise preference data from LLMs following the helpful-honest-harmless (HHH) criteria, and In-Depth Data Generation, which enriches the dataset with responses spanning a wide quality range. With HHH-guided preference data, our approach simultaneously enables the LLMs to learn human preferences and align with human values. Quantitative assessments on five benchmark datasets demonstrate the reliability and potential of AutoPM, pointing out a more general and scalable way to improve LLM performance.
BibTeX
@inproceedings{huang2023learning,
  author = {Shijia Huang and Jianqiao Zhao and Yanyang Li and Liwei Wang},
  title = {{Learning Preference Model for LLMs via Automatic Preference Data Generation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  url = {https://openreview.net/forum?id=RLmpJ4xol2},
}
Figure