MidPO: Dual Preference Optimization for Safety and Helpfulness in Large Language Models via a Mixture of Experts Framework

Yupeng Qi, Ziyu Lyu, Min Yang, Yanlin Wang, Lu Bai, Lixin Cui

TL;DR

This paper introduces MidPO, a novel Mixture of Experts (MoE) framework that effectively balances safety and helpfulness in Large Language Models (LLMs). By creating specialized "safety" and "helpfulness" experts and dynamically routing requests between them, MidPO significantly outperforms existing methods in simultaneously improving both crucial LLM capabilities.

What Problem Does This Paper Address?

Large Language Models (LLMs) are becoming increasingly powerful and integrated into various applications. However, a critical challenge is ensuring they are both helpful to users and safe, meaning they avoid generating harmful, biased, or unethical content. Existing approaches to achieve this "dual preference optimization" fall into two camps:

  • Safety-constrained online alignment: These methods can sometimes lead to "over-refusal," where the LLM becomes too cautious and refuses even safe requests, thus reducing helpfulness.
  • Safety-constrained offline alignment: These methods can struggle to adaptively balance safety and helpfulness, sometimes generating harmful content when faced with unsafe inputs.

The authors aim to bridge this gap by developing a method that can effectively and adaptively optimize for both safety and helpfulness simultaneously.

Methods

MidPO introduces a novel Mixture of Experts (MoE) framework with two core components:

  1. Single-Preference Enhanced Experts (SPE-DPO): The base LLM is transformed into two independent "experts" using a modified Direct Preference Optimization (DPO) method. One expert is fine-tuned specifically for safety, and the other for helpfulness. This SPE-DPO approach uses a "homogeneous preference margin" to amplify the distinction between preferred and dispreferred responses within each category.
  2. Dynamic Routing Mechanism: The two experts are integrated into an MoE architecture. A dynamic router, implemented as two independent linear networks, adaptively allocates contributions from each expert based on the input prompt. This allows the model to leverage the safety expert for unsafe queries and the helpfulness expert for safe ones, achieving a dynamic trade-off. The router is trained to balance these contributions using a combination of safety and helpfulness preferences.

Datasets

The paper utilizes the following datasets:

  • PKU-Safe RLHF: Used for training experts and the router. Specific subsets are used for safety-only preference data.
  • Wildguard Mix: Used for evaluation, particularly for prompts related to "harmful" categories.
  • Do Not Answer: Used for evaluation.

Key Insights and Ideas

  • Specialization is Key: Creating distinct experts for safety and helpfulness, rather than trying to optimize both simultaneously in a single model, allows for more targeted improvements.
  • Dynamic Routing is Crucial: A static approach to balancing safety and helpfulness is insufficient. A dynamic routing mechanism that can adapt to the nature of the input prompt is essential for achieving optimal performance across diverse scenarios.
  • SPE-DPO Enhances Expert Performance: The proposed SPE-DPO method effectively enhances the individual performance of each expert, leading to better specialized capabilities.
  • MoE Architecture Enables Integration: The MoE framework provides a flexible way to combine the specialized experts and leverage the dynamic routing mechanism.

Key Results

  • Superior Overall Performance: MidPO consistently outperforms baseline methods (MODPO, Safe RLHF, DPOH, DPOS) in both safety and helpfulness scores across multiple benchmark datasets.
  • High Win Rates: MidPO achieves significantly higher win rates (e.g., 86.15% for safety and 63.09% for helpfulness compared to Alpaca-7B) in human and LLM-based evaluations.
  • Effective Safety on Unsafe Inputs: MidPO demonstrates a strong ability to maintain safety even when faced with unsafe requests, unlike some baselines that generate harmful content.
  • Balanced Performance on Safe Inputs: For safe requests, MidPO avoids the "over-refusal" issue seen in some methods, maintaining good helpfulness while remaining safe.
  • Marginal Inference Time Increase: While MidPO introduces a slight increase in inference time (+0.49s compared to Safe RLHF), the significant gains in safety and helpfulness make this overhead justifiable.

Limitations and Future Work

  • Transformer Layer Choice: The study focused on fine-tuning the "down_proj" layer for LoRA. It remains an open question whether applying the framework to other linear layers would yield comparable results.
  • Extending to Other Objectives: Applying MidPO to additional alignment objectives would necessitate pretraining supplementary reward models, potentially impacting efficiency.
  • Computational Resources: The authors note that reproducing Safe RLHF was challenging due to high computational resource requirements and the lack of fully open-sourced datasets.

The authors suggest that future work could explore applying MidPO to other linear layers and extending it to more complex alignment objectives.

Summary updated

Abstract

As large language models (LLMs) are increasingly applied across various domains, enhancing safety while maintaining the helpfulness of LLMs has become a critical challenge. Recent studies solve this problem through safety-constrained online preference optimization or safety-constrained offline preference optimization. However, the safety-constrained online methods often suffer from excessive safety, which might reduce helpfulness, while the safety-constrained offline methods perform poorly in adaptively balancing safety and helpfulness. To address these limitations, we propose MidPO, a Mixture of Experts (MoE) framework for safety-helpfulness dual Preference Optimization. Firstly, MidPO devises single-preference enhanced direct preference optimization approach to transform the base model into two independent experts, termed safety and helpfulness experts, and fine-tunes the two independent experts for optimal safety or helpfulness performance. Secondly, to achieve an effective balance between safety and helpfulness, MidPO incorporates the two experts into the MoE framework and designs a dynamic routing mechanism to allocate contributions from each expert adaptively. We conduct quantitative and qualitative experiments on three popular datasets to demonstrate the proposed MidPO significantly outperforms state-of-the-art approaches in both safety and helpfulness. Code is available at https: //github.com/OutdoorManofML/MidPO.
BibTeX
@inproceedings{qi2025midpo,
  author = {Yupeng Qi and Ziyu Lyu and Min Yang and Yanlin Wang and Lu Bai and Lixin Cui},
  title = {{MidPO: Dual Preference Optimization for Safety and Helpfulness in Large Language Models via a Mixture of Experts Framework}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure