The Paradox of Preference: A Study on LLM Alignment Algorithms and Data Acquisition Methods

Rishikesh Devanathan, Varun Nathan, Ayush KumarNAACL 2024

TL;DR

This paper investigates how the source of preference data (human vs. LLM-generated) and its quality/quantity affect the performance of LLM alignment algorithms like DPO. Surprisingly, LLM-generated preferences sometimes outperform human ones, and data volume/quality showed no clear correlation with performance, challenging common assumptions in the field.

What Problem Does This Paper Address?

Aligning Large Language Models (LLMs) with human preferences is crucial for making them helpful, harmless, and honest. While Reinforcement Learning from Human Feedback (RLHF) and its variants like Direct Preference Optimization (DPO) are common, collecting high-quality human preference data is expensive and time-consuming. This has led to the exploration of using LLMs themselves to generate preference data (Reinforcement Learning from AI Feedback - RLAIF). However, the impact of using LLM-generated preferences versus human preferences, and how data volume and quality influence these alignment algorithms, remains underexplored. This paper aims to fill this gap by systematically studying these factors.

Methods

The researchers compare several LLM alignment algorithms:

  • Supervised Fine-Tuning (SFT): A baseline method.
  • Direct Preference Optimization (DPO): A single-stage method that treats preference learning as a classification task.
  • Identity Preference Optimization (IPO): An extension of DPO that introduces a regularizing term to mitigate overfitting.
  • Conservative DPO (cDPO): A variant that uses a conservative target distribution to improve stability.

They investigate the influence of:

  • Preference Source: Human vs. LLM (GPT4-Turbo) generated preferences.
  • Data Volume: Varying the percentage of preference data used.
  • Data Quality: Introducing "flip noise" (swapping chosen and rejected responses) to simulate noisy data.

Models were trained on the Zephyr-7B LLM and evaluated using the MT Bench benchmark, which assesses conversational abilities across various domains.

Datasets

  • GAIR Dataset: Used for training, containing both human and LLM (GPT4-Turbo) preferences.
  • MT Bench: Used for evaluation, a benchmark with 80 multi-turn questions rated by GPT-4.
  • Ultrafeedback Dataset: Used for additional experiments, containing LLM preferences.
  • Stanford Human Preferences Dataset (SHP): Used for additional experiments, containing human preferences.

Key Insights and Ideas

  • LLM Preferences Can Outperform Human Preferences: This is a major counter-intuitive finding. The DPO model trained on GPT4-Turbo preferences surprisingly outperformed the one trained on human preferences, challenging the notion that human preferences are always the gold standard for alignment.
  • No Clear Correlation Between Data Volume/Quality and Performance: Contrary to expectations, increasing the amount of preference data or introducing noise did not consistently lead to better or worse performance. This suggests that simply having more data or cleaner data isn't a guaranteed path to better alignment.
  • DPO's Overfitting Susceptibility is Nuanced: While previous work suggested DPO is prone to overfitting, this study found it only exhibited overfitting when trained on human preferences with a default hyperparameter setting. IPO, designed to counter overfitting, didn't consistently outperform DPO.
  • cDPO Doesn't Offer a Clear Advantage: Despite its design to be more conservative and robust to noise, cDPO did not perform better than DPO, even under noisy conditions.

Key Results

  • RQ1 (Preference Model Performance): DPO trained on GPT4-Turbo preferences achieved higher MT Bench scores than DPO trained on human preferences. Human evaluation of model outputs confirmed this, with GPT4-Turbo preference-trained models winning 63 out of 160 samples (39.4% win rate).
  • RQ2 (Data Volume): No monotonic relationship was found between data volume and performance for DPO and IPO. In some cases, models trained on less data (25% human preferences) outperformed those trained on more data, hinting at potential overfitting issues.
  • RQ3 (Data Quality): DPO models trained with 25% flip noise surprisingly outperformed those trained on clean data. cDPO consistently performed worse than DPO across all noise levels.
  • RQ4 (Overfitting): DPO showed overfitting only when trained on human preferences with a default beta value. IPO did not consistently show resilience against overfitting.

Limitations and Future Work

The authors acknowledge several limitations:

  • Algorithm Scope: The study focused on DPO, IPO, and cDPO; results might differ for other alignment algorithms.
  • Dataset Specificity: Findings are based on GAIR, Ultrafeedback, and SHP datasets and MT Bench evaluation; other datasets might yield different results.
  • Model Specificity: Zephyr-7B and GPT4-Turbo were used; other models might behave differently.
  • No Correlation Finding: The lack of correlation between data volume/quality and performance doesn't rule out other influencing factors.
  • Human Preference Validity: The unexpected superiority of LLM preferences raises questions about using human preferences as a sole benchmark.
  • Overfitting Detection: The study's methods for detecting overfitting might not have been sensitive enough, especially given the relatively small GAIR dataset size and short training epochs.

Future work could explore a wider range of alignment algorithms, different datasets and models, and delve deeper into the reasons behind the observed counter-intuitive results, particularly the superiority of LLM preferences and the lack of clear data volume/quality impact. Further research is also needed to fully understand the conditions that lead to overfitting.

Summary updated

BibTeX
@inproceedings{devanathan2024paradox,
  author = {Rishikesh Devanathan and Varun Nathan and Ayush Kumar},
  title = {{The Paradox of Preference: A Study on LLM Alignment Algorithms and Data Acquisition Methods}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure