Multi-objective Reinforcement learning from AI Feedback

Marcus WilliamsarXiv 2024 · DOI · arXiv

Abstract

Aligning Large Language Models (LLMs) with complex human values often faces a "black box" bottleneck where a single reward model struggles to capture nuanced preferences. This paper introduces Multi-Objective Reinforcement Learning from AI Feedback (MORLAIF), a modular framework that decomposes human preference into 12 distinct, manageable principles. By using AI feedback to train specialized preference models, MORLAIF achieves superior alignment and demonstrates that smaller models can effectively provide scalable oversight for larger systems.

Introduction: The RLHF Bottleneck

Reinforcement Learning from Human Feedback (RLHF) is the current industry standard for making LLMs helpful and safe. However, it relies on training a monolithic Reward Model (or Preference Model) to represent the entirety of human values.

This approach creates a significant bottleneck: human preferences are nuanced, multi-faceted, and often self-conflicting. A single model attempting to learn "what humans like" often becomes a black box susceptible to reward hacking—where the LLM finds loopholes to achieve high scores without actually improving performance. Furthermore, as models grow more capable, it becomes increasingly difficult for humans to provide the high-quality feedback necessary to train these monolithic models.

The MORLAIF Framework

MORLAIF shifts the strategy from a monolithic reward model to a "committee" of specialized ones. The process involves four key stages:

  1. Decomposition: Human preference is broken down into 12 specific principles, such as helpfulness, ethicality, factuality, toxicity, sycophancy, and conciseness.
  2. AI Feedback (RLAIF): Instead of expensive human labels, GPT-3.5-Turbo acts as a judge, evaluating pairs of responses based on these specific principles.
  3. Preference Modeling: Separate, lightweight models are trained for each principle using Low-Rank Adaptation (LoRA).
  4. Scalarization: These 12 scores are combined into a single reward signal for Reinforcement Learning (PPO) using functions like Weighted Linear, Minimax, or Bernoulli-Nash.
The MORLAIF framework: Preference Modelling (left) uses AI feedback to train specialized models, which then provide reward signals via a scalarization function during the RL loop (right).

The study utilized benchmarks such as OpenAssistant-Guanaco for supervised fine-tuning and Anthropic’s HH-RLHF for prompt evaluation across model sizes ranging from GPT-2 to Llama-7B.

Key Insights

The Decomposition Advantage

A striking finding is that the act of decomposing the problem is more significant than the mathematical method used to recombine the scores. The choice of scalarization function—whether a simple average or a complex minimax function—did not significantly alter final performance. This suggests that specificity in feedback is the primary driver of alignment quality.

Scalable Oversight and Specificity

MORLAIF demonstrates a path toward "scalable oversight." The experiments showed that a small model (GPT-2-medium) could effectively align a much larger model (Llama-7B). This suggests that ensembles of smaller, verifiable models can be used to keep frontier AI systems in check.

Furthermore, preference models trained on narrow rules (e.g., "is this toxic?") achieved higher accuracy than those trained on broad, subjective "human preference" labels. It is easier for a model to learn a narrow rule than a broad value. Notably, "sycophancy" was the only principle that negatively correlated with others, highlighting a flaw in current datasets where "helpful" responses are often just "agreeable" ones. MORLAIF allows researchers to explicitly dial down sycophancy by adjusting its weight in the reward function.

Results and Evaluation

In human evaluations, the MORLAIF-trained Llama-7B model was preferred over the standard single-objective RLAIF baseline by a significant margin.

Win rates as judged by human crowdworkers, comparing the Multi-Objective (MO) approach against Single-Objective (SO) baselines for Llama-2-7b.

Automated validation using GPT-4-Turbo confirmed these results, with MORLAIF models consistently outperforming baselines across all tested model sizes. Notably, the use of LoRA for the 12 preference models ensured that MORLAIF consumed zero extra VRAM compared to single-objective approaches during the RL phase, making it a highly efficient upgrade.

Limitations and Future Work

While MORLAIF excels for mid-sized models, the benefits appear to diminish as target models reach "frontier" scales. Additionally, the system remains dependent on the quality and biases of the initial feedback model (GPT-3.5). Future research will explore more granular principles and the potential for multi-objective framing to prevent "mode collapse" in large-scale RL training.

Reference: Williams, M. (2024). Multi-Objective Reinforcement Learning from AI Feedback. arXiv:2406.07295v2 [cs.LG].

Summary updated

Abstract

This paper presents Multi-Objective Reinforcement Learning from AI Feedback (MORLAIF), a novel approach to improving the alignment and performance of language models trained using reinforcement learning from AI feedback (RLAIF). In contrast to standard approaches that train a single preference model to represent all human preferences, MORLAIF decomposes this task into multiple simpler principles, such as toxicity, factuality, and sycophancy. Separate preference models are trained for each principle using feedback from GPT-3.5-Turbo. These preference model scores are then combined using different scalarization functions to provide a reward signal for Proximal Policy Optimization (PPO) training of the target language model. Our experiments indicate that MORLAIF outperforms the standard RLAIF baselines and that MORLAIF can be used to align larger language models using smaller ones. Surprisingly, the choice of scalarization function does not appear to significantly impact the results.
BibTeX
@misc{williams2024multi,
  author = {Marcus Williams},
  title = {{Multi-objective Reinforcement learning from AI Feedback}},
  year = {2024},
  eprint = {2406.07295},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2406.07295},
}
Figure