MidPO: Dual Preference Optimization for Safety and Helpfulness in Large Language Models via a Mixture of Experts Framework

Yupeng Qi, Ziyu Lyu, Min Yang, Yanlin Wang, Lu Bai, Lixin Cui

Abstract

As large language models (LLMs) are increasingly applied across various domains, enhancing safety while maintaining the helpfulness of LLMs has become a critical challenge. Recent studies solve this problem through safety-constrained online preference optimization or safety-constrained offline preference optimization. However, the safety-constrained online methods often suffer from excessive safety, which might reduce helpfulness, while the safety-constrained offline methods perform poorly in adaptively balancing safety and helpfulness. To address these limitations, we propose MidPO, a Mixture of Experts (MoE) framework for safety-helpfulness dual Preference Optimization. Firstly, MidPO devises single-preference enhanced direct preference optimization approach to transform the base model into two independent experts, termed safety and helpfulness experts, and fine-tunes the two independent experts for optimal safety or helpfulness performance. Secondly, to achieve an effective balance between safety and helpfulness, MidPO incorporates the two experts into the MoE framework and designs a dynamic routing mechanism to allocate contributions from each expert adaptively. We conduct quantitative and qualitative experiments on three popular datasets to demonstrate the proposed MidPO significantly outperforms state-of-the-art approaches in both safety and helpfulness.

Introduction

Aligning Large Language Models (LLMs) involves a delicate balancing act: the model must be helpful (following instructions effectively) and safe (refusing harmful requests). Current approaches generally fall into two camps, both with flaws. Safety-constrained online alignment (e.g., Safe RLHF) often suffers from "excessive safety," where the model becomes over-cautious and refuses harmless requests because it over-optimizes for safety metrics. Conversely, safety-constrained offline alignment (e.g., MODPO) often uses fixed hyper-parameters to balance the two goals, making them brittle and unable to adaptively prioritize safety or helpfulness based on the specific context of a user's prompt.

MidPO tackles this by moving away from a "one-size-fits-all" model weights approach toward a modular, expert-based architecture. By decoupling safety and helpfulness into separate experts, the model avoids the "gradient interference" that happens when trying to optimize both in a single set of weights.

Methodology

The MidPO framework consists of a two-stage optimization process designed to create and then blend specialized capabilities.

The MidPO framework architecture, illustrating the two-stage process: (a) training specialized safety and helpfulness experts via SPE-DPO, and (b) integrating them into a dynamic routing MoE mechanism.

Single-Preference Enhanced DPO (SPE-DPO)

Instead of training one model on both safety and helpfulness data simultaneously, MidPO creates two specialized experts. A "homogeneous preference margin" is introduced into the DPO loss function to amplify the distinction between "winning" and "losing" responses within a single category. This results in two distinct LoRA (Low-Rank Adaptation) experts: $E^s_\theta$ (Safety) and $E^h_\theta$ (Helpfulness).

Dynamic Routing Mechanism

MidPO integrates these LoRA experts into the model's MLP layers. A lightweight, trainable "router" examines the input hidden states and dynamically calculates weights for each expert. If a prompt is safe, the router leans toward the helpfulness expert; if it detects risk, it boosts the safety expert's contribution. The router is trained using a "reward-ranked" approach on a combined dataset, ensuring it learns the optimal "mix" for any given input while the underlying experts remain frozen.

Experimental Results

The model was evaluated using three primary datasets: PKU-Safe RLHF, Wildguard Mix, and Do Not Answer. MidPO was tested against SOTA baselines like Safe RLHF and MODPO using Alpaca-7B as the base.

Performance comparison of MidPO against state-of-the-art baselines across three datasets using both reward-model-based and LLM-based metrics. MidPO consistently achieves the best balance between safety and helpfulness scores.

MidPO achieved the highest average win rates across all datasets. In safety, it reached an average win rate of 86.15%, and in helpfulness, 63.09%, significantly outperforming Safe RLHF (79.17% safety / 51.34% helpfulness). Qualitative case studies showed that MidPO successfully answered safe prompts that Safe RLHF refused, while simultaneously providing more robust refusals for truly harmful prompts where MODPO failed.

The framework adds a marginal latency of approximately +0.49s per response compared to standard models, which is a justifiable cost for the gains in reliability and the reduction of over-refusal.

Conclusion

MidPO demonstrates that modular alignment via a Mixture of Experts framework effectively resolves the persistent trade-off between safety and helpfulness in LLMs. By using a dynamic router to act as a real-time "safety switch," the model achieves context-aware blending that outperforms static offline methods and over-cautious online methods. Future work may explore scaling this approach to more specialized experts across different domains like coding or creative writing.

Summary updated

Abstract

As large language models (LLMs) are increasingly applied across various domains, enhancing safety while maintaining the helpfulness of LLMs has become a critical challenge. Recent studies solve this problem through safety-constrained online preference optimization or safety-constrained offline preference optimization. However, the safety-constrained online methods often suffer from excessive safety, which might reduce helpfulness, while the safety-constrained offline methods perform poorly in adaptively balancing safety and helpfulness. To address these limitations, we propose MidPO, a \textbf{\underline{Mi}}xture of Experts (MoE) framework for safety-helpfulness \textbf{\underline{d}}ual \textbf{\underline{P}}reference \textbf{\underline{O}}ptimization. Firstly, MidPO devises single-preference enhanced direct preference optimization approach to transform the base model into two independent experts, termed safety and helpfulness experts, and fine-tunes the two independent experts for optimal safety or helpfulness performance. Secondly, to achieve an effective balance between safety and helpfulness, MidPO incorporates the two experts into the MoE framework and designs a dynamic routing mechanism to allocate contributions from each expert adaptively. We conduct quantitative and qualitative experiments on three popular datasets to demonstrate the proposed MidPO significantly outperforms state-of-the-art approaches in both safety and helpfulness. The code and models will be released.
BibTeX
@inproceedings{qi2025midpo,
  author = {Yupeng Qi and Ziyu Lyu and Min Yang and Yanlin Wang and Lu Bai and Lixin Cui},
  title = {{MidPO: Dual Preference Optimization for Safety and Helpfulness in Large Language Models via a Mixture of Experts Framework}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure