Bi-Factorial Preference Optimization: Balancing Safety-Helpfulness in Language Models

Wenxuan Zhang, Philip H. S. Torr, Mohamed Elhoseiny, Adel Bibi

Abstract: Fine-tuning large language models (LLMs) on human preferences, typically through reinforcement learning from human feedback (RLHF), has proven successful in enhancing their capabilities. However, ensuring the safety of LLMs during fine-tuning remains a critical concern, and mitigating the potential conflicts in safety and helpfulness is costly in RLHF. To address this issue, we propose a supervised learning framework called Bi-Factorial Preference Optimization (BFPO), which re-parameterizes a joint RLHF objective of both safety and helpfulness into a single supervised learning objective. In supervised optimization, a labeling function is used to capture the global preferences ranking to balance both safety and helpfulness. To evaluate BFPO, we develop a benchmark that includes comprehensive discriminative and generative tasks for helpfulness and harmlessness. The results indicate that our method significantly outperforms existing approaches in both safety and helpfulness. Moreover, BFPO achieves the same level of safety as methods that heavily rely on human labor with less than 10% of the computational resources and human prompting and annotation process.

The Alignment Tax: Helpfulness vs. Harmlessness

Aligning Large Language Models (LLMs) with human values typically involves navigating two competing objectives: helpfulness (providing effective answers) and harmlessness (refusing toxic or illegal content). This tension is often referred to as the "alignment tax."

Current methods, such as standard Direct Preference Optimization (DPO), frequently struggle to maintain this balance. A model optimized solely for helpfulness may provide dangerous instructions, while one over-optimized for safety might refuse benign requests, such as coding assistance, due to overly sensitive triggers. As illustrated in the introductory examples, naive combinations of safety and helpfulness datasets often result in models that are either dangerously compliant or functionally useless.

Figure 1: The tension between helpfulness and safety in different training regimes. Model (a) is helpful but unsafe; (b) is safe but unhelpful; (c) is a naive mix that fails at both. BFPO (d) achieves the optimal balance.

Methodology: Bi-Factorial Preference Optimization (BFPO)

The core innovation of BFPO is a mathematical re-parameterization that transforms a complex, multi-objective reinforcement learning problem into a single, efficient supervised learning task. This approach bypasses the stability issues and hyperparameter sensitivity typically associated with Proximal Policy Optimization (PPO).

Global Labeling Function ($g_I$)

Instead of binary "win/loss" labels, BFPO utilizes a continuous labeling function that incorporates both safety and helpfulness. This function explicitly ranks responses into a hierarchy where a safe and helpful response is the "gold standard." Crucially, a helpful but harmful response is penalized more heavily than a safe but mediocre one. This "Safety First" heuristic ensures the model prioritizes helpfulness only when the response is confirmed to be safe.

Figure 6: The conceptual framework of BFPO, mapping response types (Hateful, Desired, Neutral, Acceptable) onto a Helpfulness-Safety quadrant (left) and the corresponding global preference ranking logic used to define the labeling function (right).

Theoretical and Empirical Alignment

The authors provide theoretical proof that minimizing the supervised BFPO loss is equivalent to maximizing rewards in a multi-objective RLHF setup. By calculating the "preference gap" between potential responses using the $g_I$ function, the model learns nuanced trade-offs between being informative and remaining secure.

Experimental Results

BFPO was evaluated using a comprehensive suite of public datasets, including UltraChat and UltraFeedback for helpfulness, and PKU-SafeRLHF for safety. The evaluation benchmark included discriminative tasks for bias (BBQ) and moral reasoning (ETHICS), alongside generative tasks for toxicity (AdvBench).

Key findings include:

  • Superior Safety: On a Mistral-7B base, BFPO improved the average harmlessness score to 77.16%, a significant jump from the 49.03% achieved by standard DPO.
  • Helpfulness Preservation: While safety-only fine-tuning often causes helpfulness scores to plummet (e.g., AlpacaEval dropping to ~4%), BFPO maintained high helpfulness (AlpacaEval ~13.33%) while maximizing safety.
  • Resource Efficiency: BFPO achieves results comparable to or better than traditional safety alignment (like Llama 2) while using less than 10% of the computational and human-labeling resources.

Discussion and Limitations

While BFPO represents a significant step forward in efficient alignment, it relies on explicit safety labels within the training data. In cases where datasets lack these labels, a separate safety classifier is required. Future work may involve extending this "bi-factorial" framework to "multi-factorial" optimization, incorporating additional objectives such as honesty, conciseness, and specific formatting requirements.

Summary updated

Abstract

Fine-tuning large language models (LLMs) on human preferences, typically through reinforcement learning from human feedback (RLHF), has proven successful in enhancing their capabilities. However, ensuring the safety of LLMs during fine-tuning remains a critical concern, and mitigating the potential conflicts in safety and helpfulness is costly in RLHF. To address this issue, we propose a supervised learning framework called Bi-Factorial Preference Optimization (BFPO), which re-parameterizes a joint RLHF objective of both safety and helpfulness into a single supervised learning objective. In supervised optimization, a labeling function is used to capture the global preferences ranking to balance both safety and helpfulness. To evaluate BFPO, we develop a benchmark that includes comprehensive discriminative and generative tasks for helpfulness and harmlessness. The results indicate that our method significantly outperforms existing approaches in both safety and helpfulness. Moreover, BFPO achieves the same level of safety as methods that heavily rely on human labor with less than 10\% of the computational resources and human prompting and annotation process. The training recipes can be found here: https://github.com/wx-zhang/bfpo.
BibTeX
@inproceedings{zhang2025bi,
  author = {Wenxuan Zhang and Philip H. S. Torr and Mohamed Elhoseiny and Adel Bibi},
  title = {{Bi-Factorial Preference Optimization: Balancing Safety-Helpfulness in Language Models}},
  booktitle = {International Conference on Learning Representations},
  year = {2025},
  url = {https://openreview.net/forum?id=GjM61KRiTG},
}
Figure