Reinforced Model Merging

Jiaqi Han, Jingwen Ye, Shunyu Liu, Haofei Zhang, Jie Song, Zunlei Feng, Mingli Song

Introduction

Model merging has emerged as a powerful, training-free technique for creating versatile AI systems by combining the weights of multiple specialized models into a single multi-task architecture. By operating directly in the parameter space, researchers can fuse models—such as one specialized in bird identification and another in dog identification—without the prohibitive cost of full-scale retraining.

However, existing merging techniques face two critical challenges:

  1. The "Uniformity" Trap: Most methods apply a single merging rule (e.g., simple weight averaging) across every layer of the model. This ignores the fact that different layers capture different levels of abstraction; some layers may benefit from direct selection, while others require complex blending. This lack of flexibility often leads to "catastrophic forgetting."
  2. The Efficiency Bottleneck: While searching for optimal merging "recipes" is possible, it is traditionally extremely slow. Evaluating a merged model candidate typically requires inference over entire datasets, making large-scale search computationally expensive.
Comparison of traditional uniform merging (a) vs. Reinforced Model Merging (b), which searches for a custom layer-wise architecture.

Reinforced Model Merging (RMM)

To address these hurdles, the authors propose Reinforced Model Merging (RMM). This framework reframes the merging process as a sequential decision-making problem for a Reinforcement Learning (RL) agent. Instead of a "one-size-fits-all" approach, an agent learns to navigate the model layer-by-layer to determine the optimal operation for each specific part of the network.

The RL Framework

The system utilizes Proximal Policy Optimization (PPO) to train an agent that interacts with a "merging environment." At each layer $k$, the agent selects an action from a comprehensive Action Space:

  • Model Action: Select weights directly from a specific source model (e.g., Model A or Model B).
  • Merging Action: Apply mathematical blending operators such as Task Arithmetic or Ties-Merging.
  • Layer Action: Modify the network topology by skipping layers or returning to previous ones, allowing for the creation of non-linear, optimized architectures.
The RMM framework: An RL agent selects actions from a diverse space to optimize the merging map, using Dynamic Average Reward (DAR) for efficient evaluation.

Efficiency via Dynamic Average Reward (DAR)

The "secret sauce" behind RMM’s speed is the Dynamic Average Reward (DAR) mechanism. To avoid the bottleneck of full-dataset evaluation, RMM evaluates candidates on tiny, random subsets of data. While these small batches provide noisy signals, DAR uses a moving average to stabilize the reward, allowing the agent to learn effectively from sparse feedback. This approach enables a 96.5x speedup compared to traditional search methods, reducing hours of computation to mere minutes.

Experimental Results

The researchers evaluated RMM across diverse vision and language benchmarks, including CIFAR, CUB-200 (birds), Stanford Dogs, and various NLP datasets using ViT and T5 architectures.

  • Performance Gains: RMM consistently outperformed standard baselines like Weight Averaging and Ties-Merging. In vision tasks, it improved average accuracy by 5.57% to 8.56%.
  • Specialized Task Mastery: On challenging datasets like CUB-200, RMM achieved a massive performance jump, increasing accuracy from ~47% (standard mergers) to 69.02%.
  • Resource Efficiency: Because RMM is gradient-free and does not require backpropagation, it uses significantly less memory than traditional fine-tuning, making it suitable for execution on consumer-grade hardware.

Conclusion

Reinforced Model Merging demonstrates that heterogeneous, layer-wise merging is superior to uniform approaches. By leveraging RL and the DAR mechanism, RMM provides a path toward highly optimized multi-task models that are both high-performing and computationally efficient. Future work aims to extend this framework to heterogeneous model merging (combining different architectures) and multi-modal fusion across vision and text domains.

Summary updated

BibTeX
@inproceedings{han2025reinforced,
  author = {Jiaqi Han and Jingwen Ye and Shunyu Liu and Haofei Zhang and Jie Song and Zunlei Feng and Mingli Song},
  title = {{Reinforced Model Merging}},
  booktitle = {ICME},
  year = {2025},
  url = {https://openreview.net/forum?id=CKhEwyMlHd},
}
Figure