Mixup Model Merge: Enhancing Model Merging Performance through Randomized Linear Interpolation
Its authors withdrew this paper, so there is no full text to read or summarize.
Lists
Only you see this
Abstract
Model merging aims to integrate multiple task-specific models into a unified model that inherits the capabilities of the task-specific models without additional training. Existing model merging methods often lack consideration of the varying contribution ratios of different task-specific models to the final merged model. In this paper, we propose Mixup Model Merge ($M^3$), a simple yet effective method inspired by the randomized linear interpolation strategy from the Mixup data augmentation technique. $M^3$ performs randomized linear interpolation in parameter space between two task-specific LLMs, where interpolation coefficients are sampled from a Beta distribution to explore diverse contribution ratios. This controllable randomness allows $M^3$ to outperform standard equal-ratio merging by discovering better contribution ratio combinations. Extensive experiments show that $M^3$ significantly: improves merged LLM performance across tasks; enhances out-of-distribution and adversarial robustness; outperforms the positive effects of the sparsification method DARE on model merging and can be further combined with DARE to achieve superior results; and by tuning the Beta distribution’s shape parameters, $M^3$ balances exploration efficiency and diversity in contribution ratios. The code is provided in the supplementary materials.
BibTeX
@misc{yuezhou2025mixup,
author = {YueZhou and Yi Chang and Yuan Wu},
title = {{Mixup Model Merge: Enhancing Model Merging Performance through Randomized Linear Interpolation}},
year = {2025},
eprint = {2502.15434},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2502.15434},
}