Abstract
Fine-grained control over large language models (LLMs) remains a significant challenge, hindering their adaptability to diverse user needs. While Reinforcement Learning from Human Feedback (RLHF) shows promise in aligning LLMs, its reliance on scalar rewards often limits its ability to capture diverse user preferences in real-world applications. To address this limitation, we introduce the Directional Preference Alignment (DPA) framework. Unlike the scalar-reward RLHF, DPA incorporates multi-objective reward modeling to represent diverse preference profiles. Additionally, DPA models user preferences as directions (i.e., unit vectors) in the reward space to achieve user-dependent preference control. Our method involves training a multi-objective reward model and then fine-tuning the LLM with a preference-conditioned variant of Rejection Sampling Finetuning (RSF), an RLHF method adopted by Llama 2. This method enjoys a better performance trade-off across various reward objectives. In comparison with the scalar-reward RLHF, DPA offers users intuitive control over LLM generation: they can arithmetically specify their desired trade-offs (e.g., more helpfulness with less verbosity). We also validate the effectiveness of DPA with real-world alignment experiments on Mistral-7B. Our method provides straightforward arithmetic control over the trade-off between helpfulness and verbosity while maintaining competitive performance with strong baselines such as Direct Preference Optimization (DPO).
Introduction
Standard Reinforcement Learning from Human Feedback (RLHF) typically relies on a scalar reward model to align Large Language Models (LLMs) with human values. This approach assumes a singular "correct" behavior for any given prompt, often leading to a "one-size-fits-all" output. In practice, this frequently results in excessive verbosity, as models learn that longer responses correlate with higher reward scores in training datasets—a phenomenon known as verbosity bias.
However, human preferences are inherently diverse and context-dependent. A user seeking a quick summary has fundamentally different requirements than one seeking a detailed technical breakdown. To address this, we propose Directional Preference Alignment (DPA), a framework that enables precise, arithmetic control over LLM behavior by modeling preferences as vectors within a multi-objective reward space.
The Directional Preference Alignment (DPA) Framework
DPA moves beyond scalar rewards by implementing a three-stage alignment process:
- Multi-Objective Reward Modeling: Instead of training a model to produce a single score, we develop a reward model that outputs a vector of $k$ distinct attributes, such as helpfulness, correctness, coherence, complexity, and verbosity.
- Directional Preference Representation: We represent a user's specific trade-off preference as a unit vector $v$ in the reward space. The scalar reward for a specific response is then calculated as the dot product of the reward vector and this preference vector. This allows the system to prioritize different attributes based on the direction of the vector.
- Iterative Rejection Sampling Finetuning (RSF): The model is fine-tuned using a preference-conditioned variant of RSF. During training, the model is prompted with a specific preference direction. It generates multiple candidate responses, and the response that best aligns with the chosen direction is selected for the next round of supervised fine-tuning. This process is repeated iteratively to expand the model's Pareto front—the boundary of its optimal performance across all objectives.
Experimental Setup
We evaluated DPA using the Mistral-7B model as a base. Training and evaluation were conducted using two primary datasets:
- HelpSteer: Featuring 37k responses across 10k prompts, annotated for five specific attributes: helpfulness, correctness, coherence, complexity, and verbosity.
- UltraFeedback: A large-scale dataset (64k prompts) providing GPT-4 labels for honesty, truthfulness, instruction-following, and helpfulness.
Key Advantages of DPA
Arithmetic Control
DPA provides users with intuitive "knobs" to adjust model behavior. By prepending a preference vector to the system prompt (e.g., "maximize helpfulness by 0.8 and reduce verbosity by 0.6"), the model can dynamically shift its generation style to meet specific constraints without requiring separate fine-tuned versions for every use case.
The Feasibility Guarantee
A significant limitation of previous steerable models, such as SteerLM, is the "feasibility" problem. Asking a model to achieve an impossibly high reward score for a very short answer can lead to erratic behavior. DPA avoids this by optimizing for a direction rather than a fixed target value. The model is simply tasked with performing as well as possible along the specified vector, ensuring the request is always mathematically feasible.
Results and Discussion
Our experiments demonstrate that DPA-aligned models consistently achieve Pareto dominance over standard baselines like DPO and SteerLM. This means that for any specific verbosity level, DPA provides higher helpfulness scores than its competitors.
Furthermore, the iterative nature of DPA allows the model's capabilities to grow over time. By the fourth iteration ($t=4$), the model significantly outperformed initial Supervised Fine-Tuning (SFT) benchmarks. On AlpacaEval 2.0, DPA achieved competitive win rates against GPT-4 Turbo while maintaining significantly shorter average response lengths than DPO-trained models, effectively mitigating the verbosity trap.
Conclusion and Future Work
Directional Preference Alignment offers a robust solution to the limitations of scalar-reward RLHF. By treating preferences as directions in a multi-objective space, DPA provides users with precise arithmetic control over LLM outputs while maintaining high performance.
Future research will focus on scaling DPA to a larger number of objectives, such as safety, humor, and technical depth. Additionally, there is potential to explore the integration of directional alignment with other popular algorithms like Direct Preference Optimization (DPO) to further enhance the efficiency of the alignment process.
Summary updated