Mixup Model Merge: Enhancing Model Merging Performance through Randomized Linear Interpolation

Yue Zhou, Yi Chang, Yuan Wu

Introduction

Model merging has emerged as a powerful technique in the open-source AI community, enabling the fusion of specialized Large Language Models (LLMs) into multi-capable systems without the prohibitive costs of additional training. By blending the weights of models—for instance, one optimized for code generation and another for mathematical reasoning—researchers can create a single model that inherits the strengths of its parents. However, traditional merging methods often rely on rigid, fixed contribution ratios (typically 50/50) or require exhaustive manual tuning to find the optimal balance.

The Challenge of Fixed Ratios

Existing frameworks like Task Arithmetic and TIES-Merging generally assume that a uniform scaling factor is sufficient. In reality, the "sweet spot" for merging is rarely at the exact midpoint. One model may contain more parameter noise, or a specific task might require a disproportionate contribution from one specialized set of weights. Manually searching for these optimal ratios is computationally expensive and scales poorly with the increasing size of modern LLMs.

Methodology: Mixup Model Merge (M3)

The proposed Mixup Model Merge (M3) method addresses these limitations by introducing Randomized Linear Interpolation. Inspired by the "Mixup" data augmentation technique used in computer vision, M3 treats the merging ratio as a stochastic variable rather than a fixed constant.

The M3 process: Randomly sampling merging ratios from a Beta distribution to explore the parameter space between two models.

The core of M3 involves sampling a merging coefficient $\lambda_m$ from a Beta distribution: $$\theta_M = \lambda_m \theta_1 + (1 - \lambda_m) \theta_2$$ where $\lambda_m \sim \text{Beta}(\alpha, \alpha)$. The hyperparameter $\alpha$ controls the distribution's shape:

  • $\alpha < 1$: The distribution is bimodal, favoring ratios that lean heavily toward one of the parent models.
  • $\alpha > 1$: The distribution peaks in the middle, favoring a more balanced mix.
  • $\alpha = 1$: Represents a uniform distribution across all possible ratios.

M3 is designed as a "plug-and-play" enhancement for existing merging frameworks. By sampling a small number of ratios (typically seven), M3 can efficiently identify a merged model that significantly outperforms standard baseline configurations.

Experimental Evaluation

The authors evaluated M3 using 13B-parameter models based on Llama-2: WizardLM (instruction following), WizardMath (mathematical reasoning), and Code-Alpaca (code generation). Performance was measured across diverse benchmarks, including AlpacaEval for instructions, GSM8K and MATH for reasoning, and HumanEval and MBPP for coding. To test robustness, the authors utilized LiveBench for out-of-distribution (OOD) tasks and PromptBench for adversarial attacks.

Key Results and Insights

M3 demonstrated substantial improvements across all tested benchmarks and merging frameworks.

Performance comparison of merged models using Task Arithmetic (TA) and TIES-Merging with and without M3 across various benchmarks.

Performance and Robustness

  • Substantial Gains: Integrating M3 with Average Merging for a Math & Code blend resulted in a 7.43% increase in GSM8K scores and an 11.0% boost in MBPP scores compared to standard 50/50 merging.
  • Adversarial Resilience: M3-merged models showed remarkable robustness. In a notable case involving the CoLA dataset, the Performance Drop Rate (PDR) under adversarial attack was reduced from 98.53% to just 6.42%.
  • OOD Stability: M3 consistently outperformed non-randomized baselines on LiveBench, suggesting that the randomized search leads to better generalization on unseen tasks.
  • Recovering Performance: As seen in the benchmarks, M3 can even "rescue" merging methods like TIES-Merging in scenarios where they would otherwise fail (e.g., HumanEval and MBPP), effectively compensating for weaknesses in individual models.

Theoretical Justification

The success of M3 is supported by the "Low-Loss Basin" theory, which posits that models fine-tuned from the same base initialization reside within a shared, flat region of the loss landscape. Linear interpolation allows for movement between these models without encountering high-error "valleys." By introducing randomness, M3 effectively explores this shared space to find optimal parameter configurations that a human might overlook.

Conclusion

Mixup Model Merge (M3) provides a simple yet effective strategy for optimizing LLM blending. By replacing fixed ratios with randomized sampling from a Beta distribution, M3 discovers superior model combinations that enhance performance, robustness, and generalization. Future work may extend this approach to multi-model merging using Dirichlet distributions and explore its application in mitigating the "alignment tax" during Reinforcement Learning from Human Feedback (RLHF).

Summary updated

Abstract

Model merging aims to integrate multiple task-specific models into a unified model that inherits the capabilities of the task-specific models without additional training. Existing model merging methods often lack consideration of the varying contribution ratios of different task-specific models to the final merged model. In this paper, we propose Mixup Model Merge ($M^3$), a simple yet effective method inspired by the randomized linear interpolation strategy from the Mixup data augmentation technique. $M^3$ performs randomized linear interpolation in parameter space between two task-specific LLMs, where interpolation coefficients are sampled from a Beta distribution to explore diverse contribution ratios. This controllable randomness allows $M^3$ to outperform standard equal-ratio merging by discovering better contribution ratio combinations. Extensive experiments show that $M^3$ significantly: improves merged LLM performance across tasks; enhances out-of-distribution and adversarial robustness; outperforms the positive effects of the sparsification method DARE on model merging and can be further combined with DARE to achieve superior results; and by tuning the Beta distribution’s shape parameters, $M^3$ balances exploration efficiency and diversity in contribution ratios. The code is provided in the supplementary materials.
BibTeX
@misc{zhou2025mixup,
  author = {Yue Zhou and Yi Chang and Yuan Wu},
  title = {{Mixup Model Merge: Enhancing Model Merging Performance through Randomized Linear Interpolation}},
  year = {2025},
  eprint = {2502.15434},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2502.15434},
}
Figure