Bi-Factorial Preference Optimization: Balancing Safety-Helpfulness in Language Models

Wenxuan Zhang, Philip Torr, Mohamed Elhoseiny, Adel Bibi

In the world of Large Language Models (LLMs), developers face a constant tug-of-war known as the "alignment tension." On one side, we want models to be as helpful as possible, following instructions to the letter. On the other, we need them to be safe, refusing to generate harmful, biased, or illegal content.

BFPO vs DPO on Safety and Helpfulness

Traditionally, balancing these two goals requires Reinforcement Learning from Human Feedback (RLHF), a complex and expensive process involving multiple reward models and massive amounts of human-annotated "red-teaming" data. A new paper from researchers at KAUST and Oxford proposes a much more efficient alternative: Bi-Factorial Preference Optimization (BFPO).

TL;DR

BFPO is a supervised learning framework that mathematically merges the goals of safety and helpfulness into a single optimization objective. By using a clever labeling function to rank responses, it achieves state-of-the-art safety and helpfulness with less than 10% of the computational and human resources typically required by traditional RLHF methods.

The Problem: The Alignment Tension

Aligning an LLM is usually a two-step dance: first, make it helpful; then, make it safe. However, these objectives often conflict. A "perfectly safe" model might become a "refusal machine," declining even harmless prompts (over-refusal). Conversely, a "highly helpful" model might accidentally provide instructions for dangerous activities.

Current state-of-the-art methods use multi-objective RLHF. This involves training separate reward models for safety and helpfulness and then combining them. This process is notoriously difficult to tune, computationally heavy, and requires thousands of hours of human labor to generate specific "red-teaming" prompts where the model is likely to fail.

Methods: Bi-Factorial Preference Optimization

The core innovation of BFPO is the re-parameterization of the multi-reward RLHF objective into a single supervised learning objective, building on the foundations of Direct Preference Optimization (DPO).

The Secret Sauce: The Labeling Function ($g_I$)

Instead of just telling the model which of two answers is "better," BFPO uses a specialized labeling function to capture a global preference ranking.

The BFPO Labeling Function and Response Quadrant

As shown above, responses are mapped onto a quadrant of Helpfulness (Desired vs. Acceptable) and Safety (Safe vs. Harmful/Hateful). BFPO categorizes pairs of responses into four distinct classes:

  1. Safe (Desired) vs. Unsafe (Neutral): Prefer the helpful, safe one.
  2. Safe (Desired) vs. Safe (Acceptable): Prefer the more helpful one.
  3. Unsafe (Hateful) vs. Unsafe (Neutral): Neutral or negative (we don't want either).
  4. Unsafe (Hateful) vs. Safe (Acceptable): Strongly prefer the safe one.

By assigning specific numerical weights to these scenarios using a shifting factor ($\alpha$), the model learns a nuanced hierarchy: Safe + Helpful > Safe + Unhelpful > Unsafe. The paper provides a mathematical proof showing that minimizing this supervised BFPO loss is theoretically equivalent to maximizing the combined rewards in a traditional multi-objective RL setting.

Key Insights and Results

The researchers tested BFPO against standard DPO, IPO, and multi-objective RL (MORL) using a Mistral-7B base, utilizing datasets like UltraChat, UltraFeedback, and PKU-SafeRLHF.

  • Global Ranking over Binary Choice: Safety and helpfulness shouldn't be treated as separate scores to be added together. Instead, they form a global hierarchy where safety acts as a gatekeeper for helpfulness.
  • The Power of the Shifting Factor ($\alpha$): A small shift in the safety label value helps the model distinguish between "bad" and "worse" unsafe responses, significantly reducing the likelihood of generating harmful content.
  • Superior Safety & Maintained Helpfulness: BFPO achieved a generative harmlessness score of 95.24%, vastly outperforming standard DPO (~40%). Unlike other safety-focused methods that cause the model's utility to crash, BFPO maintained an AlpacaEval score comparable to models trained purely for instruction following.
  • Resource Efficiency: BFPO achieves the safety levels of Llama 2-class models using only 30K data points and a single iteration of training.

Limitations and Future Work

While highly effective, BFPO currently relies on datasets having explicit binary safety labels. If a dataset only has "A is better than B" without saying why, BFPO's current labeling function can't be fully utilized. Furthermore, future research is needed to see if this "bi-factorial" approach can scale to "multi-factorial" optimization (e.g., adding Honesty, Conciseness, and Reasoning as separate factors).

Check out the code and training recipes: https://github.com/wx-zhang/bfpo

Summary updated

Abstract

Fine-tuning large language models (LLMs) on human preferences, typically through reinforcement learning from human feedback (RLHF), has proven successful in enhancing their capabilities. However, ensuring the safety of LLMs during fine-tuning remains a critical concern, and mitigating the potential conflicts in safety and helpfulness is costly in RLHF. To address this issue, we propose a supervised learning framework called Bi-Factorial Preference Optimization (BFPO), which re-parameterizes a joint RLHF objective of both safety and helpfulness into a single supervised learning objective. In the supervised optimization, a labeling function is used to capture global preferences ranking to balance both safety and helpfulness. To evaluate BFPO, we develop a benchmark including comprehensive discriminative and generative tasks for helpfulness and harmlessness. The results indicate that our method significantly outperforms existing approaches in both safety and helpfulness. Moreover, BFPO eliminates the need for human prompting and annotation in LLM fine-tuning while achieving the same level of safety as methods that heavily rely on human labor, with less than 10\% of the computational resources. The training recipes and models will be released.
BibTeX
@inproceedings{zhang2025bi,
  author = {Wenxuan Zhang and Philip Torr and Mohamed Elhoseiny and Adel Bibi},
  title = {{Bi-Factorial Preference Optimization: Balancing Safety-Helpfulness in Language Models}},
  booktitle = {International Conference on Learning Representations},
  year = {2025},
  url = {https://openreview.net/forum?id=GjM61KRiTG},
}
Figure