TL;DR
This paper introduces Expectation Preference Optimization (EPO), a novel method for improving Large Language Model (LLM) reasoning capabilities. EPO addresses limitations in existing preference optimization techniques like DPO by leveraging groups of responses instead of single pairs, leading to more reliable preference estimation and superior performance on reasoning tasks.
What Problem Does This Paper Address?
Large Language Models (LLMs) have shown impressive capabilities in various tasks, including mathematical and commonsense reasoning, often through supervised fine-tuning (SFT). However, the performance of SFT is capped by the quality and quantity of training data, which can be expensive to create. Preference optimization methods, like Direct Preference Optimization (DPO), have emerged as a way to improve LLMs using human feedback or other preference signals. While DPO is effective for aligning LLMs with human values, it struggles with complex reasoning tasks where preferences are multifaceted. In these scenarios, sampling a single preferred and dispreferred response can lead to inaccurate preference estimations and suboptimal model updates. This paper aims to bridge this gap by proposing a method that can more reliably estimate preferences in complex reasoning scenarios, thereby enhancing LLM reasoning abilities without requiring additional human annotations.
Methods
The core contribution of this paper is the Expectation Preference Optimization (EPO) algorithm. EPO builds upon the Bradley-Terry model, which is fundamental to preference learning. Unlike DPO, which uses a single pair of preferred and dispreferred responses, EPO operates on groups of responses.
Here's how it works:
- Group-wise Preference: For a given prompt, EPO considers a group of generated responses. It then defines the "reward" of a group as the expectation of rewards for individual responses within that group.
- Expectation Calculation: This group-wise expectation allows EPO to estimate preferences more robustly. The paper derives a new objective function for EPO based on this expectation, which is shown to be equivalent to the optimal solution of the RLHF objective.
- Self-Improvement Framework: EPO is integrated into a self-improvement training paradigm. In this framework, an SFT model generates multiple responses for a given query. These responses are then categorized into a "preferred group" (Gw) if they provide the correct answer and a "dispreferred group" (Gι) if they provide an incorrect answer. EPO then uses these groups to update the model.
- Length Limitation: To further stabilize the optimization process, especially for dispreferred responses, EPO incorporates a length limitation operation, truncating responses to a preset threshold.
EPO is presented as a variant of DPO, and importantly, when the group size is reduced to one, EPO effectively becomes DPO.
Datasets
The paper evaluates EPO on several reasoning benchmarks:
- Arithmetic Reasoning:
- GSM8K: A widely used dataset for mathematical reasoning.
- MetaMaths: An augmentation of GSM8K and MATH datasets.
- MathQA: Contains math queries with multiple-choice answers and rationales.
- Commonsense Reasoning:
- ARC (AI2 Reasoning Challenge): Consists of ARC-Easy and ARC-Challenge subsets.
- SocialIQA: Focuses on commonsense reasoning about social interactions.
The authors also describe prompts used for generating rationales and for analyzing false positive samples, but these are not presented as new datasets.
Key Insights and Ideas
- Beyond Pairwise: The fundamental insight is that for complex reasoning tasks, relying on single preference pairs (like in DPO) is insufficient. Grouping responses and considering their collective "expectation" provides a more stable and accurate signal for preference learning.
- Reasoning vs. Alignment: The paper highlights that while DPO is effective for human value alignment (where preferences are often clearer), it falters in reasoning tasks where errors can be multifaceted and less clear-cut. EPO's group-based approach is better suited for this complexity.
- Self-Improvement Synergy: EPO integrates seamlessly into self-improvement frameworks, allowing LLMs to refine their reasoning abilities using their own generated outputs, thus reducing reliance on external annotations.
- Length Limitation's Role: The length limitation module, while seemingly simple, is shown to contribute to optimization stability, particularly in the context of reasoning tasks.
Key Results
EPO consistently outperforms various baseline methods (SFT, RFT, DPO, DPObatch, RPO, Step-DPO) across multiple reasoning benchmarks and base LLMs (Llama3-8B, Llama2-13B, Qwen2.5-7B, Mistral-7B).
- Math Reasoning: EPO achieved significant accuracy improvements, for example, a 5.43% increase over SFT on GSM8K for Llama2-13B.
- Commonsense Reasoning: Similar gains were observed, such as a 3.58% increase for Llama3-8B on SocialIQA.
- Robustness: EPO demonstrates more stable performance compared to DPO and DPObatch, which can sometimes degrade model performance.
- Efficiency: While EPO requires processing larger input tensors (groups of responses), its training time is only about 2-3 times that of other methods (for N < 30) due to efficient parallelization, with only a small increase in GPU memory cost.
Limitations and Future Work
- Alignment Tasks: The paper acknowledges that while EPO's theory is general, applying it to alignment tasks (where answers aren't strictly binary correct/incorrect) requires finding a proper method to calculate expectation in such contexts.
- Future Directions: The authors suggest exploring further methods to enhance EPO, such as incorporating weights for responses within groups, to further refine preference estimation.
Summary updated