Introduction
Model merging has emerged as a powerful technique in the open-source AI community, enabling the fusion of specialized Large Language Models (LLMs) into multi-capable systems without the prohibitive costs of additional training. By blending the weights of models—for instance, one optimized for code generation and another for mathematical reasoning—researchers can create a single model that inherits the strengths of its parents. However, traditional merging methods often rely on rigid, fixed contribution ratios (typically 50/50) or require exhaustive manual tuning to find the optimal balance.
The Challenge of Fixed Ratios
Existing frameworks like Task Arithmetic and TIES-Merging generally assume that a uniform scaling factor is sufficient. In reality, the "sweet spot" for merging is rarely at the exact midpoint. One model may contain more parameter noise, or a specific task might require a disproportionate contribution from one specialized set of weights. Manually searching for these optimal ratios is computationally expensive and scales poorly with the increasing size of modern LLMs.
Methodology: Mixup Model Merge (M3)
The proposed Mixup Model Merge (M3) method addresses these limitations by introducing Randomized Linear Interpolation. Inspired by the "Mixup" data augmentation technique used in computer vision, M3 treats the merging ratio as a stochastic variable rather than a fixed constant.
The core of M3 involves sampling a merging coefficient $\lambda_m$ from a Beta distribution: $$\theta_M = \lambda_m \theta_1 + (1 - \lambda_m) \theta_2$$ where $\lambda_m \sim \text{Beta}(\alpha, \alpha)$. The hyperparameter $\alpha$ controls the distribution's shape:
- $\alpha < 1$: The distribution is bimodal, favoring ratios that lean heavily toward one of the parent models.
- $\alpha > 1$: The distribution peaks in the middle, favoring a more balanced mix.
- $\alpha = 1$: Represents a uniform distribution across all possible ratios.
M3 is designed as a "plug-and-play" enhancement for existing merging frameworks. By sampling a small number of ratios (typically seven), M3 can efficiently identify a merged model that significantly outperforms standard baseline configurations.
Experimental Evaluation
The authors evaluated M3 using 13B-parameter models based on Llama-2: WizardLM (instruction following), WizardMath (mathematical reasoning), and Code-Alpaca (code generation). Performance was measured across diverse benchmarks, including AlpacaEval for instructions, GSM8K and MATH for reasoning, and HumanEval and MBPP for coding. To test robustness, the authors utilized LiveBench for out-of-distribution (OOD) tasks and PromptBench for adversarial attacks.
Key Results and Insights
M3 demonstrated substantial improvements across all tested benchmarks and merging frameworks.
Performance and Robustness
- Substantial Gains: Integrating M3 with Average Merging for a Math & Code blend resulted in a 7.43% increase in GSM8K scores and an 11.0% boost in MBPP scores compared to standard 50/50 merging.
- Adversarial Resilience: M3-merged models showed remarkable robustness. In a notable case involving the CoLA dataset, the Performance Drop Rate (PDR) under adversarial attack was reduced from 98.53% to just 6.42%.
- OOD Stability: M3 consistently outperformed non-randomized baselines on LiveBench, suggesting that the randomized search leads to better generalization on unseen tasks.
- Recovering Performance: As seen in the benchmarks, M3 can even "rescue" merging methods like TIES-Merging in scenarios where they would otherwise fail (e.g., HumanEval and MBPP), effectively compensating for weaknesses in individual models.
Theoretical Justification
The success of M3 is supported by the "Low-Loss Basin" theory, which posits that models fine-tuned from the same base initialization reside within a shared, flat region of the loss landscape. Linear interpolation allows for movement between these models without encountering high-error "valleys." By introducing randomness, M3 effectively explores this shared space to find optimal parameter configurations that a human might overlook.
Conclusion
Mixup Model Merge (M3) provides a simple yet effective strategy for optimizing LLM blending. By replacing fixed ratios with randomized sampling from a Beta distribution, M3 discovers superior model combinations that enhance performance, robustness, and generalization. Future work may extend this approach to multi-model merging using Dirichlet distributions and explore its application in mitigating the "alignment tax" during Reinforcement Learning from Human Feedback (RLHF).
Summary updated