Abstract
Despite the advanced capacities of state-of-the-art large language models (LLMs), they suffer from issues of hallucination, stereotype, and misalignment. Preference models (PMs) play an important role in LLM alignment, yet training them predominantly relies on human-annotated data, which limits versatility and scalability. In this paper, we propose AutoPM, a framework for learning preference models via automatic preference data generation. Our approach involves both In-Breadth Data Generation, which elicits pairwise preference data following helpful-honest-harmless (HHH) criteria, and In-Depth Data Generation, which enriches the dataset with responses spanning a wide quality range. Quantitative assessments demonstrate that AutoPM enables LLMs to learn human preferences and align with human values effectively, offering a scalable path to improved LLM performance.
Introduction
Reinforcement Learning from Human Feedback (RLHF) has become the standard for aligning LLMs with human values. This process relies on a Preference Model (PM) that ranks responses to guide the policy model. Traditionally, training these PMs requires thousands of human-labeled comparisons—a process that is slow, expensive, and difficult to scale. While AI Feedback (using models like GPT-4 to rank responses) has been explored, existing methods often lack a systematic framework to ensure data diversity and adherence to safety guidelines. AutoPM addresses this by using a "guideline-driven" approach to automatically generate 42k preference pairs, bypassing the human annotation bottleneck.
Methodology
AutoPM generates synthetic preference data through three distinct strategies designed to capture both broad alignment principles and fine-grained quality nuances.
1. In-Breadth Data Generation
To ensure the model understands the "HHH" criteria, the framework uses pairs of opposite attributes (e.g., "useful" vs. "useless," "fact-based" vs. "fabricated"). An LLM (GPT-3.5) is prompted to generate two responses for the same instruction: one following the positive attribute and one following the negative one. This creates a clear signal for the PM to distinguish between helpful and unhelpful behavior.
2. In-Depth Data Generation
To prevent the PM from becoming a simple binary classifier, AutoPM creates a quality gradient. Starting with a high-quality response, the system applies specific Edit Actions to degrade it:
- Deletion: Removing useful content.
- Substitution: Changing content to be inaccurate.
- Insertion: Adding irrelevant "noise." This forces the PM to learn subtle differences in response quality rather than just surface-level patterns.
3. Paired Models Data
Finally, the framework leverages the inherent performance gap between models of different scales. Responses from a strong model (GPT-3.5) serve as "chosen" examples, while responses from a smaller model (OPT-IML 1.3B) serve as "rejected" examples for the same prompt.
Experimental Results
The final preference model was trained by fine-tuning a LLaMA-7B backbone on the 42k-sample synthetic dataset. Evaluation was conducted across several benchmarks, including HHH-Eval, TruthfulQA, and the Vicuna benchmark.
Performance and Consistency
AutoPM-7B achieved an overall accuracy of 0.83 on HHH-Eval, nearly matching the 0.86 score of Anthropic’s 52B parameter model trained on massive human datasets. Crucially, AutoPM’s rankings showed high consistency with both human annotators and GPT-4, outperforming other open-source preference models like PandaLM.
Alignment via Rejection Sampling
The study demonstrates that AutoPM can improve existing models without complex Reinforcement Learning (PPO). By generating multiple responses and using AutoPM to select the best one (Rejection Sampling), the truthfulness of models like Alpaca and Vicuna was significantly boosted. For instance, Alpaca-7B’s TruthfulQA score rose from 0.38 to 0.51.
Key Insights
- Generalization: Even though AutoPM was trained on data from GPT-3.5, it successfully discriminated between responses from stronger models like GPT-4. This suggests the PM learns fundamental principles of preference rather than just mimicking a specific model's style.
- Negative Constraints: Explicitly prompting the generator to be "useless" or "dishonest" provides a much stronger training signal than random sampling.
Conclusion and Future Work
AutoPM provides a scalable, automated alternative to human-centric alignment. While an expert review found minor noise in the synthetic data (1-2% error rate), the model's performance remains robust. Future research will explore more complex edit actions and the potential for a "self-evolving" alignment loop where models iteratively improve their own preference data.
Summary updated