Preference-Guided Reflective Sampling for Aligning Language Models

Hai Ye, Hwee Tou Ng

Introduction

Aligning Large Language Models (LLMs) with human expectations is a cornerstone of modern AI development, typically involving iterative cycles of data generation and Reinforcement Learning from Human Feedback (RLHF). The industry standard for generating training data is Repeated Random Sampling (or "Best-of-N"), where a model generates multiple independent responses and a reward model selects the best one.

However, this approach suffers from significant inefficiencies: it relies on pure randomness to explore the output space, lacks the ability to learn from previous attempts during sampling, and struggles to incorporate specific, diverse natural language preferences (e.g., "be concise" or "include references"). To address these flaws, we introduce Preference-Guided Reflective Sampling (PRS), a tree-based sampling method that leverages explicit preferences and self-reflective feedback loops to generate higher-quality alignment data.

Methods: The PRS Framework

PRS moves away from independent random samples toward a structured, tree-based generation process. By incorporating natural language preferences as an explicit part of the sampling context, PRS guides the model toward high-reward zones more intelligently than random chance.

The PRS workflow combines natural language preferences with a tree-based generation structure to iteratively refine model responses.

The PRS Workflow:

  1. Initial Exploration ($N_0$): The model generates a set of initial responses to a prompt.
  2. Selection (Exploitation): A reward model identifies the highest-performing initial response ($y^*_0$).
  3. Reflective Feedback: The model acts as its own critic, generating natural language feedback ($f$) based on a specific user preference ($z$) to improve the selected response.
  4. Guided Refinement ($N_1$): The model generates a new set of "refinement" samples, conditioned on the original prompt, the best previous response, and the new feedback.
  5. Iterative Depth: This process can be repeated across multiple layers of a tree, allowing the model to progressively hone in on optimal outputs.

Key Insights

  • Sustained Improvement: Standard random sampling often causes model improvement to plateau after one iteration of RL training. PRS generates higher-quality data in every cycle, enabling continuous improvement across multiple training iterations.
  • Exploration vs. Exploitation: The tree structure formally balances these needs: the root ($N_0$) provides broad exploration, while subsequent refinement layers ($N_1, N_2...$) provide exploitation by searching the neighborhood of the best-known answer.
  • Self-Correction as a Tool: While often used for single-output improvement, self-correction is most powerful as a data generation tool for creating superior training sets for offline RL.

Key Results

PRS was evaluated across diverse tasks including instruction following, summarization, and safety. On standard "vibe-check" benchmarks, PRS consistently outperformed the random sampling baseline at the same computational budget (Best-of-32).

PRS significantly outperforms repeated random sampling (Rand) on both AlpacaEval v2.0 and Arena-Hard v0.1 benchmarks.
  • Benchmark Superiority: On AlpacaEval v2.0, PRS (using Llama-3-8b) achieved a length-controlled win rate of 36.70%, compared to 32.94% for random sampling. On Arena-Hard, PRS reached a 72.2% win rate vs. 68.2% for the baseline.
  • Safety and Versatility: By adding a "safe response" preference, PRS reduced toxicity in the ToxiGen dataset from 3.9% to 0.2%. It also outperformed established models like Vicuna-13B in keyword-focused summarization tasks.

Limitations and Future Work

While effective, PRS faces bottlenecks in extremely difficult reasoning tasks (like complex math) where the model may struggle to generate useful self-feedback. Additionally, like all alignment methods, it remains susceptible to "reward hacking." Future research will explore deeper tree structures and the integration of PRS with other alignment techniques like Direct Preference Optimization (DPO).

Summary updated

BibTeX
@inproceedings{ye2024preference,
  author = {Hai Ye and Hwee Tou Ng},
  title = {{Preference-Guided Reflective Sampling for Aligning Language Models}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.emnlp-main.1206},
}
Figure