DogeRM: Equipping Reward Models with Domain Knowledge through Model Merging

Tzu-Han Lin, Chen-An Li, Hung-yi Lee, Yun-Nung Chen

Abstract: Reinforcement learning from human feedback (RLHF) is a popular strategy for aligning large language models (LLMs) with desired behaviors. Reward modeling is a crucial step in RLHF. However, collecting paired preference data for training reward models is often costly and time-consuming, especially for domain-specific preferences requiring expert annotation. To address this challenge, we propose the Domain knowledge merged Reward Model (DogeRM), a novel framework that integrates domain-specific knowledge into a general reward model by model merging. The experiments demonstrate that DogeRM enhances performance across different benchmarks and provide a detailed analysis showcasing the effects of model merging, showing the great potential of facilitating model alignment.

Introduction

In the standard Reinforcement Learning from Human Feedback (RLHF) pipeline, the Reward Model (RM) serves as the critical judge that guides an AI toward helpful and safe responses. However, building effective RMs for specialized domains—such as advanced mathematics or low-level systems programming—presents a significant bottleneck. Training these models typically requires "preference data," where humans rank multiple AI responses. For specialized fields, this requires expert annotators, making data collection prohibitively expensive and slow.

A new framework, DogeRM, addresses this challenge by leveraging the abundance of domain-specific demonstration data (good examples of math or code) rather than rare preference data. By using model merging, DogeRM injects the deep knowledge of a specialized language model into a general-purpose reward model.

The DogeRM Framework

The core innovation of DogeRM is its training-free (or low-training) approach to domain adaptation. Instead of fine-tuning a reward model on scarce expert rankings, the framework merges two distinct models:

  1. A General RM: Trained on broad preference datasets (like UltraFeedback) to understand the general structure of a "good" response.
  2. A Domain SFT Model: A standard language model fine-tuned on domain-specific demonstrations (e.g., MetaMath for mathematics or specialized code models).
Overview of the DogeRM framework, illustrating the merging of a general Reward Model (RM) with a Domain-specific Supervised Fine-Tuning (SFT) model.

As illustrated in the framework overview, both models are decomposed into their embedding layers and transformer backbones. Parameters are combined using weighted averaging, while the specialized regression head of the Reward Model—designed to output a scalar score—is preserved. This allows the resulting "Domain RM" to utilize the domain expertise of the SFT model while retaining the evaluative capabilities of the original RM.

Key Results and Insights

The researchers evaluated DogeRM across several rigorous benchmarks, including RewardBench and Auto-J Eval, as well as practical Best-of-N sampling scenarios on GSM8K (math) and MBPP (coding).

  • Math and Coding Mastery: Merging a LLaMA-2 RM with the MAmmoTH-7B model improved math performance on RewardBench by 17% absolute (from 68.2% to 85.2%). Coding scores saw a boost of 5.4%.
  • Architecture Agnostic: The method proved highly effective across different base models. When applied to Mistral-based architectures, math performance on RewardBench jumped by a massive 30%.
  • Knowledge as a Proxy for Preference: A striking finding is that a model doesn't necessarily need to be explicitly told that a correct math proof is better than an incorrect one if it already possesses the internal knowledge to solve the problem. The SFT model's internal representation of correctness naturally enhances the RM's ability to rank responses.

Analysis: The Goldilocks Zone

A critical component of the merging process is the weight factor ($\lambda$), which determines the balance between the general RM and the domain-specific SFT model.

Impact of the merge weight λ on model accuracy, highlighting the optimal "Goldilocks Zone" for performance.

Experimental analysis reveals a "Goldilocks Zone" for $\lambda$, typically between 0.2 and 0.5. If the weight is too low, the model lacks the necessary domain expertise to judge complex tasks. However, if $\lambda$ exceeds 0.5, the model begins to lose its "reward-ness"—it forgets how to act as a judge and attempts to revert to a generative language model, leading to a sharp decline in accuracy.

Conclusion

DogeRM demonstrates that model merging is a powerful, cost-effective alternative to expert data collection for domain-specific alignment. By successfully blending general evaluative skills with deep domain knowledge, DogeRM sets a new precedent for how we can efficiently equip AI judges with the expertise needed for specialized fields. Future work will look toward integrating these merged models into full RLHF training loops to produce superior final policy models.

Summary updated

Abstract

Reinforcement learning from human feedback (RLHF) is a popular strategy for aligning large language models (LLMs) with desired behaviors. Reward modeling is a crucial step in RLHF. However, collecting paired preference data for training reward models is often costly and time-consuming, especially for domain-specific preferences requiring expert annotation. To address this challenge, we propose the **Do**main knowled**ge** merged **R**eward **M**odel (**DogeRM**), a novel framework that integrates domain-specific knowledge into a general reward model by model merging. The experiments demonstrate that DogeRM enhances performance across different benchmarks and provide a detailed analysis showcasing the effects of model merging, showing the great potential of facilitating model alignment.
BibTeX
@inproceedings{lin2024dogerm,
  author = {Tzu-Han Lin and Chen-An Li and Hung-yi Lee and Yun-Nung Chen},
  title = {{DogeRM: Equipping Reward Models with Domain Knowledge through Model Merging}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  url = {https://openreview.net/forum?id=19nRzsHhOR},
}
Figure