Annotation-Efficient Preference Optimization for Language Model Alignment

Yuu Jinnai, Ukyo Honda

Abstract

Aligning Large Language Models (LLMs) with human values typically requires extensive preference datasets, which are costly and time-consuming to produce. This paper introduces Annotation-Efficient Preference Optimization (AEPO), a method designed to maximize the utility of a limited annotation budget. AEPO strategically selects a diverse and representative subset of model-generated responses for human labeling. By focusing on high-signal pairs, AEPO significantly improves alignment performance without increasing the number of required annotations. Experiments across various benchmarks, including AlpacaFarm and specialized cultural datasets, demonstrate that AEPO outperforms standard sampling techniques and scales effectively with the number of candidate responses.

Introduction

Techniques such as Reinforcement Learning from Human Feedback (RLHF) and Direct Preference Optimization (DPO) have become standard for ensuring LLMs are helpful and harmless. These methods rely on "preference pairs"—data indicating a human's choice between two model-generated responses. However, the primary bottleneck in this process is the annotation budget. Human experts are expensive, and while AI-based labeling (RLAIF) is faster, it often introduces its own biases. To address this, the authors propose a fundamental question: If only a limited number of pairs can be labeled, which ones provide the most informative signal for model alignment?

Methodology: Quality Over Quantity

AEPO introduces a filtering and selection step prior to the annotation process. Instead of random selection or the "West-of-N" strategy—which requires labeling all $N$ generated responses to identify extremes—AEPO selects a small subset ($k$) of responses based on two primary heuristics:

  1. Representativeness ($f_{rep}$): This selects responses that are central to the distribution of all generated outputs, helping the model learn the most likely behaviors of the current LLM.
  2. Diversity ($f_{div}$): This ensures that the selected responses are distinct from one another. Labeling nearly identical responses provides redundant information and a weak training signal.

The selection objective is defined as the maximization of $f_{rep} + \lambda f_{div}$. AEPO utilizes sentence embeddings (e.g., Sentence BERT) to calculate distances between responses in a high-dimensional space, drawing inspiration from Minimum Bayes Risk (MBR) decoding.

Overview of the AEPO workflow: 1) Generate candidates, 2) Select a diverse/representative subset, 3) Annotate preferences.

Experimental Setup

The effectiveness of AEPO was evaluated on several key benchmarks:

  • AlpacaFarm: A simulation framework for instruction-following evaluation.
  • Anthropic’s hh-rlhf: A dataset focused on helpfulness and harmlessness.
  • JCommonsenseMorality (JCM): A dataset for Japanese cultural commonsense, representing a "low-resource" scenario where human-aligned data is scarce and culturally specific.

Key Results and Insights

AEPO consistently outperformed baselines such as random sampling, Coreset selection, and Perplexity-based filtering under fixed annotation budgets.

Scaling with Compute, Not Labor

A significant finding is that AEPO's performance improves as the number of candidate responses ($N$) increases, even when the number of labeled responses ($k$) remains constant. This allows researchers to use relatively inexpensive compute (generating more text) to enhance the effectiveness of expensive human labor.

Superiority Over Baselines and West-of-N

The popular "West-of-N" strategy, which picks the best and worst from $N$ labeled samples, often degrades in performance if the total number of instructions must be reduced to fit a budget. In contrast, AEPO scales gracefully. On the AlpacaFarm benchmark, AEPO achieved significantly higher win rates than West-of-N, especially as the pool of candidate responses grew to $N=128$.

Win rates on AlpacaFarm comparing AEPO against various baselines including Random sampling, West-of-N (WoN), Coreset, and Perplexity filtering.

Discussion and Limitations

While AEPO offers a path toward more efficient alignment, it has certain dependencies. The method relies heavily on the quality of the underlying embedding model to accurately measure diversity and representativeness. Additionally, while it reduces annotation costs, it increases inference costs during the candidate generation phase. Future work may explore the integration of AEPO with strategies for handling noisy or unreliable human annotators and further refining the balance between diversity and representativeness.

Summary updated

Abstract

Preference optimization is a standard approach to fine-tuning large language models to align with human preferences. The quality, diversity, and quantity of the preference dataset are critical to the effectiveness of preference optimization. However, obtaining a large amount of high-quality and diverse preference annotations is difficult in many applications. This raises the question of how to use the limited annotation budget to create an effective preference dataset. To this end, we propose Annotation-Efficient Preference Optimization (AEPO). Instead of exhaustively annotating preference over all available response texts, AEPO selects a subset of responses that maximizes quality and diversity from the available responses, and then annotates preference over the selected ones. In this way, AEPO focuses the annotation budget on labeling preference over a smaller subset of responses with diversity and of high quality. We evaluate the performance of Direct Preference Optimization (DPO) using AEPO and show that it outperforms models trained using a standard DPO with the same annotation budget. Our code is available at https://github.com/CyberAgentAILab/annotation-efficient-po
BibTeX
@misc{jinnai2024annotation,
  author = {Yuu Jinnai and Ukyo Honda},
  title = {{Annotation-Efficient Preference Optimization for Language Model Alignment}},
  year = {2024},
  eprint = {2405.13541},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2405.13541},
}
Figure