Abstract
Aligning Large Language Models (LLMs) with human values typically requires extensive preference datasets, which are costly and time-consuming to produce. This paper introduces Annotation-Efficient Preference Optimization (AEPO), a method designed to maximize the utility of a limited annotation budget. AEPO strategically selects a diverse and representative subset of model-generated responses for human labeling. By focusing on high-signal pairs, AEPO significantly improves alignment performance without increasing the number of required annotations. Experiments across various benchmarks, including AlpacaFarm and specialized cultural datasets, demonstrate that AEPO outperforms standard sampling techniques and scales effectively with the number of candidate responses.
Introduction
Techniques such as Reinforcement Learning from Human Feedback (RLHF) and Direct Preference Optimization (DPO) have become standard for ensuring LLMs are helpful and harmless. These methods rely on "preference pairs"—data indicating a human's choice between two model-generated responses. However, the primary bottleneck in this process is the annotation budget. Human experts are expensive, and while AI-based labeling (RLAIF) is faster, it often introduces its own biases. To address this, the authors propose a fundamental question: If only a limited number of pairs can be labeled, which ones provide the most informative signal for model alignment?
Methodology: Quality Over Quantity
AEPO introduces a filtering and selection step prior to the annotation process. Instead of random selection or the "West-of-N" strategy—which requires labeling all $N$ generated responses to identify extremes—AEPO selects a small subset ($k$) of responses based on two primary heuristics:
- Representativeness ($f_{rep}$): This selects responses that are central to the distribution of all generated outputs, helping the model learn the most likely behaviors of the current LLM.
- Diversity ($f_{div}$): This ensures that the selected responses are distinct from one another. Labeling nearly identical responses provides redundant information and a weak training signal.
The selection objective is defined as the maximization of $f_{rep} + \lambda f_{div}$. AEPO utilizes sentence embeddings (e.g., Sentence BERT) to calculate distances between responses in a high-dimensional space, drawing inspiration from Minimum Bayes Risk (MBR) decoding.
Experimental Setup
The effectiveness of AEPO was evaluated on several key benchmarks:
- AlpacaFarm: A simulation framework for instruction-following evaluation.
- Anthropic’s hh-rlhf: A dataset focused on helpfulness and harmlessness.
- JCommonsenseMorality (JCM): A dataset for Japanese cultural commonsense, representing a "low-resource" scenario where human-aligned data is scarce and culturally specific.
Key Results and Insights
AEPO consistently outperformed baselines such as random sampling, Coreset selection, and Perplexity-based filtering under fixed annotation budgets.
Scaling with Compute, Not Labor
A significant finding is that AEPO's performance improves as the number of candidate responses ($N$) increases, even when the number of labeled responses ($k$) remains constant. This allows researchers to use relatively inexpensive compute (generating more text) to enhance the effectiveness of expensive human labor.
Superiority Over Baselines and West-of-N
The popular "West-of-N" strategy, which picks the best and worst from $N$ labeled samples, often degrades in performance if the total number of instructions must be reduced to fit a budget. In contrast, AEPO scales gracefully. On the AlpacaFarm benchmark, AEPO achieved significantly higher win rates than West-of-N, especially as the pool of candidate responses grew to $N=128$.
Discussion and Limitations
While AEPO offers a path toward more efficient alignment, it has certain dependencies. The method relies heavily on the quality of the underlying embedding model to accurately measure diversity and representativeness. Additionally, while it reduces annotation costs, it increases inference costs during the candidate generation phase. Future work may explore the integration of AEPO with strategies for handling noisy or unreliable human annotators and further refining the balance between diversity and representativeness.
Summary updated