FusionBench: A Comprehensive Benchmark of Deep Model Fusion

Anke Tang, Li Shen, Yong Luo, Han Hu, Bo Du, Dacheng TaoSubmission to ICLR 2025 · Rejected

TL;DR

FusionBench is the first comprehensive, modular benchmark designed to systematically evaluate deep model fusion techniques across computer vision and NLP. Spanning 74 fine-tuned models, 19 fusion algorithms, and diverse evaluation settings, it establishes a unified taxonomy (ensembling, merging, and mixing) and reveals key trade-offs in multi-task transfer, robustness against distribution shifts, and generalization to unseen tasks.

What Problem Does This Paper Address?

Deep model fusion—combining the parameters or predictions of multiple task-specific models into a unified architecture—has emerged as an efficient alternative to training multi-task foundation models from scratch. However, the sub-field lacks standardized evaluation benchmarks. Existing studies typically evaluate new algorithms on bespoke, inconsistent testbeds, use disparate fine-tuning configurations (full weights vs. parameter-efficient LoRA), and rarely test fused models against distribution shifts or unseen tasks. FusionBench fills this void with a modular platform to evaluate fusion algorithms under standardized, reproducible conditions.

Methods

The authors structure model fusion into a three-part taxonomy:

  1. Model Ensemble: Combining model predictions at inference (e.g., Simple/Weighted Ensembles, Max-Model Predictor). Effective, but carries high inference and memory overhead.
  2. Model Merging: Combining parameters of isomorphic architectures into a single model of identical size without extra inference cost (e.g., Weight Averaging / Model Soups, Fisher Merging, RegMean, Task Arithmetic, TIES-Merging, and test-time adaptive methods like AdaMerging).
  3. Model Mixing: Recombining or upscaling components into a new, often heterogeneous or sparse architecture (e.g., Sparse MoE upcycling, SMILE, Weight-Ensemble MoE [WEMoE], and depth upscaling).

The open-source framework is decoupled into three primary configurable modules managed via a YAML-driven CLI:

  • Algorithm Module: Encapsulates fusion algorithms as self-contained Python classes.
  • Model Pool Module: Standardizes checkpoint loading, metadata tagging, and adaptation hooks.
  • Task Pool Module: Manages evaluation datasets, corruptions, and task-specific metrics.

Datasets

FusionBench evaluates cross-modal capabilities across several standard benchmark suites:

  • Open-Vocabulary Image Classification: Evaluated on 8 datasets (SUN397, Stanford Cars, RESISC45, EuroSAT, SVHN, GTSRB, MNIST, DTD) using CLIP-ViT-B/32, CLIP-ViT-B/16, and CLIP-ViT-L/14 backbones. Includes 7 corrupted test distributions (motion blur, impulse/Gaussian noise, pixelation, spatter, contrast, JPEG compression) based on ImageNet-C protocols.
  • Scene Understanding: 3 dense-prediction tasks (13-class segmentation, depth estimation, and surface normal estimation) on the indoor NYUv2 dataset with ResNet-50.
  • Text Classification & Text-to-Text Generation: GLUE tasks (CoLA, MNLI, MRPC, QNLI, QQP, RTE, SST-2, STSB) utilizing GPT-2, Flan-T5-Base, and Flan-T5-Large (with and without LoRA).
  • LLM Reasoning & Knowledge: Fusing 7B models (Mistral-7B derivatives) evaluated on MMLU, GSM8K, TruthfulQA, and ARC Challenge.

Key Insights and Ideas

  • Task Vectors are Nearly Orthogonal: Analyzing cosine similarity across fine-tuned CLIP task vectors revealed values close to zero (often $\le 0.05$). This empirical near-orthogonality explains why linear parameter addition (e.g., Task Arithmetic) works, but also shows that interference happens in subtle, shared subspaces.
  • Adaptive Merging and MoE Mixing Outperform Static Averages: Methods dynamically routing weights or adapting parameters at test time (AdaMerging, WEMoE, SMILE) consistently beat static weight averaging. Dynamic routing preserves distinct task capabilities that parameter averaging washes out.
  • Smaller Models Gain More from Fusion: Fusing CLIP-ViT-B/32 models showed a wider performance gap relative to single-task learning (STL) than CLIP-ViT-L/14 models. Larger pre-trained representations already capture broad cross-task features, leaving less room for linear multi-task gains.
  • Severe Vulnerability to Corruptions and Negative Transfer: When tested on unseen tasks or severe image corruptions (like impulse noise or pixelation), merged models can suffer severe performance drops—sometimes performing worse than the original zero-shot pre-trained base model (negative transfer). Test-time adaptive methods also risk overfitting to corrupted evaluation inputs.

Key Results

  • CLIP-ViT-B/32 8-Task Merging: Individual fine-tuned models averaged 90.3% accuracy, while the pre-trained base achieved 48.2%. Simple weight averaging reached only 66.5%. Advanced merging methods like Layer-Wise AdaMerging reached 82.6%, and model mixing methods reached 89.3% (SMILE) and 89.2% (WEMoE), approaching the traditional multi-task learning (MTL) upper bound (88.6%).
  • Flan-T5-Base Multi-Task Merging: Starting from a pre-trained average of 75.7% across 8 GLUE tasks, simple averaging achieved 78.2%, while SMILE model mixing matched individual LoRA fine-tuning performance closely (84.0% vs. 84.6% individual average).
  • LLM Upscaling (Mistral-7B): Merging MetaMath, Dolphin-2.1, and Speechless-Code via SMILE dense-expert upscaling (11.2B total parameters) achieved 70.66% on GSM8K and 54.23% on TruthfulQA, outperforming the individual source models on balanced multi-discipline benchmarks without additional training.

Limitations and Future Work

  • Primary Focus on Multi-Task Fusion: While the codebase implements ensembling and mixing algorithms intended for wider use cases, the current systematic benchmark focuses almost exclusively on multi-task model fusion.
  • Evaluation Cost for Frontier LLMs: Due to the high computational barrier of LLM evaluations, large language model benchmark suites are currently evaluated via external harnesses rather than natively inside the core benchmark pipelines.
  • Future Directions: The authors plan to extend FusionBench to RLHF/human preference alignment fusion, multimodal foundation model combinations, and continuous reinforcement learning scenarios.

Summary updated

Abstract

Deep model fusion is an emerging technique that unifies the predictions or parameters of several deep neural networks into a single model in a cost-effective and data-efficient manner. This enables the unified model to take advantage of the original models' strengths, potentially exceeding their performance. Although a variety of deep model fusion techniques have been introduced, their evaluations tend to be inconsistent and often inadequate to validate their effectiveness and robustness against distribution shifts. To address this issue, we introduce FusionBench, which is the first comprehensive benchmark dedicated to deep model fusion. FusionBench covers a wide range of tasks, including open-vocabulary image classification, text classification, and text-to-text generation. Each category includes up to eight tasks with corresponding task-specific models, featuring both full fine-tuning and LoRA fine-tuning, as well as models of different sizes, to ensure fair and balanced comparisons of various multi-task model fusion techniques across different tasks, model scales, and fine-tuning strategies. We implement and evaluate a broad spectrum of deep model fusion techniques. These techniques range from model ensemble methods, which combine the predictions to improve the overall performance, to model merging, which integrates different models into a single one, and model mixing methods, which upscale or recombine the components of the original models. FusionBench now contains a range of CV and NLP tasks, 74 fine-tuned models, and 19 fusion techniques, and we are committed to consistently expanding the benchmark with more tasks, models, and fusion techniques. In addition, we offer a well-documented set of resources and guidelines to aid researchers in understanding and replicating the benchmark results. This includes detailed documentation, code examples, and tutorials, making FusionBench a user-friendly and accessible platform for both beginners and experienced researchers.
BibTeX
@misc{tang2024fusionbench,
  author = {Anke Tang and Li Shen and Yong Luo and Han Hu and Bo Du and Dacheng Tao},
  title = {{FusionBench: A Comprehensive Benchmark of Deep Model Fusion}},
  year = {2024},
  eprint = {2406.03280},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2406.03280},
}
Figure