Abstract
Model merging offers a cost-effective alternative to retraining by blending specialized pretrained models into a single multitask system. However, traditional methods often focus solely on parameter weights and fail to address the diverse trade-offs required by different users. We present HM3 (Hierarchical Multi-Objective Model Merging), a framework that optimizes both model parameters and architecture. By modeling layer selection as a reinforcement learning problem and employing multi-objective optimization, HM3 produces a Pareto front of models, allowing for tailored performance across tasks like translation, reasoning, and coding.
Introduction
The rapid proliferation of fine-tuned models has made model merging a popular strategy for developing multitask AI. Instead of the prohibitive expense of training from scratch, researchers blend models specialized for distinct domains—such as math or coding—into a unified architecture.
Despite its popularity, current merging techniques face two primary challenges:
- The Parameter-Only Trap: Most methods focus exclusively on blending weights within a fixed structure, neglecting the "architecture space." Reordering or skipping layers from different source models can often yield superior results.
- The "Average" Problem: Standard merging typically seeks a single compromise model. However, user priorities vary; a legal translator may prioritize linguistic precision over mathematical logic. Current methods lack the flexibility to navigate these performance trade-offs.
Methodology
HM3 addresses these issues through a hierarchical approach that operates at both the parameter and architecture levels.
Parameter-Level Merging
At the base level, the framework generates weight vectors representing different task priorities. It utilizes advanced techniques like DARE (sparsifying weights to minimize interference) and Ties Merging (resolving sign conflicts) to create an initial set of blended parameters for each priority configuration.
Architecture-Level Merging
The core innovation of HM3 lies in its treatment of layer selection as a Markov Decision Process (MDP). Using Proximal Policy Optimization (PPO), a reinforcement learning agent learns an optimal "inference path."
- Dynamic Inference Paths: Rather than a static stack, the model learns to dynamically "jump" between layers of different source models.
- Reward and Penalty: The RL agent is rewarded for performance on target tasks, while a path complexity penalty ensures the resulting architecture remains efficient.
Multi-Objective Optimization
By iterating this process across various weight distributions, HM3 identifies a Pareto front. This set of optimal models represents the best possible trade-offs between competing tasks, providing a "menu" of specialized configurations for different user needs.
Evaluation and Results
The authors evaluated HM3 using 7B-parameter Llama-2 variants (Chat, WizardMath, and CodeLlama) and ViT-L/14 for vision tasks. Evaluation spanned several domains:
- Translation: WMT14, WMT16, and IWSLT2017 (using the chrf metric).
- Reasoning: GSM8K (Math, flexible match) and HumanEval (Code, pass@100).
- Computer Vision: Five diverse datasets including DTD (textures) and SVHN (house numbers).
Key Findings
HM3 significantly outperformed state-of-the-art merging methods like Task Arithmetic and DARE+Ties.
- Multitask Performance: On a combined benchmark, HM3 achieved scores of 44.68 (Translation), 45.62 (Math), and 43.62 (Code), far exceeding the 33.93, 38.20, and 28.20 reached by DARE+Ties.
- Pareto Quality: The Hypervolume (HV) metric, which measures the quality of the Pareto front, was 1.6824 for HM3, compared to 1.2486 when parameter-level optimization was omitted.
- Vision Domain: HM3 improved average accuracy across five vision datasets to 80.30%, up from 74.03% for standard Task Arithmetic.
Discussion
The success of HM3 highlights that architecture is a hidden lever in model merging. Even with optimized weights, the sequence and selection of layers are critical for performance. Furthermore, the RL-based strategy proves to be a reusable asset; once trained, the policy networks can predict merging strategies for various task weights without starting from scratch.
Limitations and Future Work
While effective, training the RL policy network (approx. 1000 episodes) requires significant computational resources. Future research will explore whether these hierarchical benefits scale to 70B+ parameter models and investigate more complex, non-sequential inference paths.
Summary updated