DogeRM: Equipping Reward Models with Domain Knowledge through Model Merging

Tzu-Han Lin, Chen-An Li, Hung-yi Lee, Yun-Nung Chen

Abstract

Reinforcement learning from human feedback (RLHF) is a popular strategy for aligning large language models (LLMs) with desired behaviors. Reward modeling is a crucial step in RLHF. However, collecting paired preference data for training reward models is often costly and time-consuming, especially for domain-specific preferences requiring expert annotation. To address this challenge, we propose the Domain knowledge merged Reward Model (DogeRM), a novel framework that integrates domain-specific knowledge into a general reward model by model merging. The experiments demonstrate that DogeRM enhances performance across different benchmarks and provide a detailed analysis showcasing the effects of model merging, showing the great potential of facilitating model alignment.

Introduction

In the development of Large Language Models (LLMs), Reinforcement Learning from Human Feedback (RLHF) serves as the primary mechanism for alignment. A central component of this process is the Reward Model (RM), which evaluates and scores model responses to guide the learning process. However, training effective RMs typically requires extensive paired preference data. While general conversational data is relatively accessible, specialized domains such as mathematics or programming require expert human annotators, making data collection prohibitively expensive and slow.

To circumvent this bottleneck, we introduce DogeRM (Domain knowledge merged Reward Model). This framework leverages existing domain-expert models—typically developed through Supervised Fine-Tuning (SFT)—and integrates their knowledge directly into a general-purpose Reward Model via model merging. This approach allows the RM to inherit the factual accuracy and reasoning capabilities of expert models without requiring additional domain-specific preference labels.

Methodology

The DogeRM framework utilizes a weighted parameter averaging technique to combine a general-purpose Reward Model with a domain-specific SFT model. The process is designed to be efficient and does not require further training on preference data.

Overview of the DogeRM framework, merging a general Reward Model with a domain-specific SFT model.

The merging process follows these key steps:

  1. Model Selection: A general RM and a domain-specific SFT model (e.g., a math-specialized model) are selected. Both models must share the same base architecture, such as LLaMA-2 or Mistral.
  2. Parameter Merging: The embedding layers and transformer blocks of both models are merged using weighted averaging.
  3. Reward Head Retention: Since SFT models lack a regression layer for scoring, the original Reward Model's head is retained to output the final scalar reward.
  4. Hyperparameter Tuning: A weight parameter, $\lambda$, controls the influence of the expert model. Empirical results suggest an optimal $\lambda$ of approximately 0.35, balancing expert knowledge with general judgment capabilities.

Experimental Setup

The effectiveness of DogeRM was evaluated using several base models and domain experts:

  • Base Models: AlpacaFarm (SFT) and UltraFeedback (general RM).
  • Domain Experts: Models trained on MetaMathQA, MathInstruct, OSS-Instruct, and Magicoder-Evol-Instruct.
  • Evaluation Benchmarks: RewardBench, Auto-J Eval, and Best-of-N sampling performance on GSM8K (math) and MBPP (coding).

Results and Discussion

The core finding of this work is that a Reward Model's evaluative performance is fundamentally constrained by its underlying knowledge. By merging with an SFT model, the RM gains the factual grounding necessary to distinguish between correct and incorrect complex reasoning.

Performance comparison in Best-of-N sampling for math and code domains.

Key performance highlights include:

  • Reasoning Accuracy: On RewardBench, merging with the MAmmoTH-7B model resulted in a 17% improvement in the reasoning category.
  • Coding Proficiency: Merging with specialized code models improved reward accuracy by 5.4% on RewardBench and 6% on Auto-J Eval.
  • Best-of-N Sampling: In practical applications where the RM selects the best of $N$ generated answers, DogeRM improved final accuracy on the GSM8K benchmark by 5% in a "Best-of-16" setting.
  • Generalization: The method proved effective across different architectures (LLaMA-2 and Mistral) and demonstrated the ability to create "polymath" models by merging multiple domain experts simultaneously.

Conclusion and Future Work

DogeRM presents a scalable and cost-effective alternative to traditional domain-specific reward modeling. By repurposing existing SFT experts, we can significantly enhance the alignment of LLMs in technical fields.

Future research will address current limitations, including the requirement for identical base architectures and the exploration of more sophisticated merging techniques like TIES-Merge or DARE. Additionally, expanding the framework to "softer" domains such as law or ethics and testing on larger scale models (70B+) remains a priority for future investigation.

Summary updated

Abstract

Reinforcement learning from human feedback (RLHF) is a popular strategy for aligning large language models (LLMs) with desired behaviors. Reward modeling is a crucial step in RLHF. However, collecting paired preference data for training reward models is often costly and time-consuming, especially for domain-specific preferences requiring expert annotation. To address this challenge, we propose the \textbf{Do}main knowled\textbf{ge} merged \textbf{R}eward \textbf{M}odel (DogeRM), a novel framework that integrates domain-specific knowledge into a general reward model by model merging. The experiments demonstrate that DogeRM enhances performance across different benchmarks and provide a detailed analysis showcasing the effects of model merging, showing the great potential of facilitating model alignment.
BibTeX
@inproceedings{lin2024dogerm,
  author = {Tzu-Han Lin and Chen-An Li and Hung-yi Lee and Yun-Nung Chen},
  title = {{DogeRM: Equipping Reward Models with Domain Knowledge through Model Merging}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  url = {https://openreview.net/forum?id=19nRzsHhOR},
}
Figure