Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewards

Haoxiang Wang, Yong Lin, Wei Xiong, Rui Yang, Shizhe Diao, Shuang Qiu, Han Zhao, Tong Zhang

Abstract

Fine-grained control over large language models (LLMs) remains a significant challenge, hindering their adaptability to diverse user needs. While Reinforcement Learning from Human Feedback (RLHF) shows promise in aligning LLMs, its reliance on scalar rewards often limits its ability to capture diverse user preferences in real-world applications. To address this limitation, we introduce the Directional Preference Alignment (DPA) framework. Unlike the scalar-reward RLHF, DPA incorporates multi-objective reward modeling to represent diverse preference profiles. Additionally, DPA models user preferences as directions (i.e., unit vectors) in the reward space to achieve user-dependent preference control. Our method involves training a multi-objective reward model and then fine-tuning the LLM with a preference-conditioned variant of Rejection Sampling Finetuning (RSF), an RLHF method adopted by Llama 2. This method enjoys a better performance trade-off across various reward objectives. In comparison with the scalar-reward RLHF, DPA offers users intuitive control over LLM generation: they can arithmetically specify their desired trade-offs (e.g., more helpfulness with less verbosity). We also validate the effectiveness of DPA with real-world alignment experiments on Mistral-7B. Our method provides straightforward arithmetic control over the trade-off between helpfulness and verbosity while maintaining competitive performance with strong baselines such as Direct Preference Optimization (DPO).

Introduction

Standard Reinforcement Learning from Human Feedback (RLHF) often collapses diverse human preferences into a single "average" scalar reward. This approach frequently results in undesirable behaviors like verbosity bias, where models learn that longer responses generally score higher, even when they are unnecessary. Because human preferences are multidimensional and often contradictory—where one user may desire a detailed explanation and another a concise summary—a single scalar value is insufficient for real-world alignment.

Directional Preference Alignment (DPA)

To overcome these limitations, we propose Directional Preference Alignment (DPA). This framework treats user preferences as unit vectors in a multi-objective reward space, allowing users to "arithmetically" tune trade-offs (e.g., helpfulness vs. conciseness) at inference time using a single model.

Comparison between Traditional RLHF and Directional Preference Alignment (DPA).

Methodology

The DPA framework follows a three-step pipeline:

  1. Multi-Objective Reward Modeling: Instead of a single score, we train a reward model to predict multiple attributes simultaneously, such as helpfulness, correctness, coherence, and verbosity.
  2. Directional Preference Representation: A user’s preference is represented as a unit vector $v$ in the reward space. The total reward is calculated as the dot product of this preference vector and the multi-objective reward vector. This enables "arithmetic control," where a user can specify weights like "0.8 helpfulness and -0.6 verbosity" to obtain a concise but useful answer.
  3. Iterative Rejection-Sampling Finetuning (RSF): We use an iterative approach to train the LLM. For a given prompt and a randomly sampled preference vector, we generate multiple responses and select the one that best matches the chosen direction. The LLM is then finetuned on these samples, conditioned on the preference vector provided in the system prompt.

Experimental Setup

We validated DPA using several primary datasets:

  • HelpSteer: 10K prompts with 37K responses annotated across five attributes: helpfulness, correctness, coherence, complexity, and verbosity.
  • UltraFeedback: 64K prompts with GPT-4 labels for honesty, truthfulness, instruction-following, and helpfulness.
  • BeaverTails-30K: Used for experiments involving safety and "harmlessness" as an additional objective.

Key Results and Insights

DPA allows a single model to traverse the Pareto front—the set of optimal trade-offs where one objective cannot be improved without degrading another. By embedding numerical vectors directly into the system prompt, the model learns to associate specific directions in the reward space with stylistic changes in text generation.

Arithmetic control in action: DPA allows users to specify trade-offs directly in the system prompt.

Our experiments on Mistral-7B yielded several key findings:

  • Superior Trade-offs: DPA consistently Pareto-dominated other methods, providing more helpful responses for any given level of verbosity compared to DPO or SteerLM.
  • Effective Control: In AlpacaEval-2.0, DPA demonstrated the ability to reduce response length (from ~1800 to ~400 tokens) while maintaining a competitive win rate against GPT-4-turbo.
  • Generalization: DPA models showed strong performance on RewardBench across Chat, Reasoning, and Safety categories, proving that directional control does not compromise the model's core capabilities.

Limitations and Future Work

The effectiveness of DPA is closely tied to the accuracy of the multi-objective reward model; any inherent biases in the reward model will be inherited by the LLM. Furthermore, while DPA provides better control, it cannot fulfill physically impossible requests (e.g., a perfectly helpful answer in only two words). Future work will investigate how this directional approach scales as the number of simultaneous objectives increases.

Summary updated

Abstract

Fine-grained control over large language models (LLMs) remains a significant challenge, hindering their adaptability to diverse user needs. While Reinforcement Learning from Human Feedback (RLHF) shows promise in aligning LLMs, its reliance on scalar rewards often limits its ability to capture diverse user preferences in real-world applications. To address this limitation, we introduce the Directional Preference Alignment (DPA) framework. Unlike the scalar-reward RLHF, DPA incorporates multi-objective reward modeling to represent diverse preference profiles. Additionally, DPA models user preferences as directions (i.e., unit vectors) in the reward space to achieve user-dependent preference control. Our method involves training a multi-objective reward model and then fine-tuning the LLM with a preference-conditioned variant of Rejection Sampling Finetuning (RSF), an RLHF method adopted by Llama 2. This method enjoys a better performance trade-off across various reward objectives. In comparison with the scalar-reward RLHF, DPA offers users intuitive control over LLM generation: they can arithmetically specify their desired trade-offs (e.g., more helpfulness with less verbosity). We also validate the effectiveness of DPA with real-world alignment experiments on Mistral-7B. Our method provides straightforward arithmetic control over the trade-off between helpfulness and verbosity while maintaining competitive performance with strong baselines such as Direct Preference Optimization (DPO).
BibTeX
@inproceedings{wang2024arithmetic,
  author = {Haoxiang Wang and Yong Lin and Wei Xiong and Rui Yang and Shizhe Diao and Shuang Qiu and Han Zhao and Tong Zhang},
  title = {{Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewards}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  url = {https://openreview.net/forum?id=H5QCjhkxN0},
}
Figure