Preference-Guided Reflective Sampling for Aligning Language Models

Hai Ye, Hwee Tou Ng

Abstract

Preference-Guided Reflective Sampling (PRS) is an innovative tree-based sampling strategy designed to enhance the alignment of Large Language Models (LLMs) with human preferences. Unlike traditional "Best-of-N" random sampling, PRS incorporates natural language feedback and iterative self-refinement. This approach generates higher-quality training data, leading to significant performance improvements across major benchmarks such as AlpacaEval and Arena-Hard.

Introduction

Aligning Large Language Models (LLMs) with human expectations is a critical step in their development, typically achieved through Reinforcement Learning from Human Feedback (RLHF). A central challenge in this process is the generation of high-quality "gold standard" responses for training. The current industry standard, Repeated Random Sampling (or Best-of-N), involves generating $N$ independent responses and selecting the best one based on a Reward Model (RM). However, this "blind" exploration is often computationally inefficient and fails to adapt to specific, nuanced user preferences, such as requests for conciseness or technical depth.

Methodology: Preference-Guided Reflective Sampling (PRS)

To address the limitations of random sampling, the authors propose Preference-Guided Reflective Sampling (PRS). This method transforms the sampling process from independent attempts into a structured, tree-based exploration-exploitation framework.

Comparison between Repeated Random Sampling (Rand) and Preference-Guided Reflective Sampling (PRS)

The PRS workflow consists of four primary stages:

  1. Initial Sampling ($N_0$): The model generates a set of initial responses based on the prompt and an explicit natural language preference (e.g., "I prefer a concise answer").
  2. Selection & Feedback: A Reward Model identifies the highest-scoring response. The LLM then generates natural language feedback ($f$) explaining why the response did not perfectly satisfy the stated preference.
  3. Refinement ($N_1$): Guided by the previous best response and the linguistic feedback, the model generates a new set of "refined" responses.
  4. Iterative Training: The resulting high-quality data is utilized in an offline RL training loop to fine-tune the model's policy.

By using feedback as a "compass," PRS leverages the model's reflective capabilities to bridge the gap between scalar rewards and actionable improvement.

Experimental Setup and Datasets

The efficacy of PRS was evaluated across diverse tasks and benchmarks:

  • Instruction Following: Utilizing ShareGPT for supervised fine-tuning (SFT) and Alpaca-GPT4 prompts for RL training.
  • Summarization: Testing on the CNN/Daily Mail dataset to evaluate keyword-focused summarization.
  • Safety: Evaluating toxicity reduction using the ToxiGen benchmark.
  • Evaluation Benchmarks: Performance was primarily measured via AlpacaEval v2.0 and Arena-Hard v0.1.

Key Results and Insights

PRS demonstrated substantial gains over traditional sampling methods across all tested benchmarks. By providing the model with a clear path for improvement through feedback, PRS allows for continued performance gains where random sampling typically plateaus.

Performance of PRS vs. Random Sampling on AlpacaEval v2.0 and Arena-Hard v0.1
  • Benchmark Performance: On AlpacaEval v2.0, PRS achieved a 36.70% win rate, outperforming the 32.94% achieved by random sampling at the same computational budget (Best-of-32). Similarly, on Arena-Hard, PRS reached a 72.2% win rate compared to 68.2% for the baseline.
  • Toxicity Mitigation: PRS significantly improved safety, reducing toxicity in the ToxiGen benchmark to 0.2%, a sharp decrease from the 3.9% observed with standard random sampling.
  • Optimal Resource Allocation: The study found that a balanced 50/50 split between initial sampling ($N/2$) and refinement ($N/2$) generally yields the best results, preventing the model from getting stuck in narrow local optima.

Discussion and Limitations

While PRS offers a powerful tool for alignment, it is not without limitations. The quality of the self-refinement feedback is dependent on the model's reasoning capabilities; for highly complex logical or mathematical tasks, the feedback may be less reliable. Additionally, like all RLHF methods, PRS remains susceptible to "reward hacking," where models optimize for the Reward Model's preferences rather than true human utility. Finally, the generation of feedback and refinements increases the token count per training sample, though the authors argue this cost is offset by the superior data quality.

Conclusion

Preference-Guided Reflective Sampling represents a significant step forward in LLM alignment. By moving from blind random sampling to a preference-guided, reflective process, PRS enables more efficient exploration of the output space and produces models that are both higher-performing and more controllable. Future work will explore the integration of PRS with techniques like Direct Preference Optimization (DPO) and its application to specialized reasoning domains.

Summary updated

BibTeX
@inproceedings{ye2024preference,
  author = {Hai Ye and Hwee Tou Ng},
  title = {{Preference-Guided Reflective Sampling for Aligning Language Models}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.emnlp-main.1206},
}
Figure