FusionBench: A Comprehensive Benchmark of Deep Model Fusion

Anke Tang, Li Shen, Yong Luo, Han Hu, Bo Du, Dacheng Tao

Abstract

Deep model fusion is an emerging technique that unifies the predictions or parameters of several deep neural networks into a single model in a cost-effective and data-efficient manner. This enables the unified model to take advantage of the original models' strengths, potentially exceeding their performance. Although a variety of deep model fusion techniques have been introduced, their evaluations tend to be inconsistent and often inadequate to validate their effectiveness and robustness against distribution shifts. To address this issue, we introduce FusionBench, the first comprehensive benchmark dedicated to deep model fusion. FusionBench covers a wide range of tasks, including open-vocabulary image classification, text classification, and text-to-text generation. Each category includes up to eight tasks with corresponding task-specific models, featuring both full fine-tuning and LoRA fine-tuning, as well as models of different sizes, to ensure fair and balanced comparisons of various multi-task model fusion techniques. FusionBench currently contains 26 distinct tasks, 74 fine-tuned models, and 16 fusion techniques.

Introduction

In the current "learn from model" era of AI, we are no longer just training networks from scratch; we are increasingly building new models by standing on the shoulders of existing ones. Deep model fusion—the art of combining multiple neural networks into a single, more capable unit—has exploded in popularity. As the community produces thousands of fine-tuned models (like those on Hugging Face), researchers have developed various ways to "fuse" them to create multi-task "Swiss Army knives." You might merge the weights of two Large Language Models (LLMs) or ensemble the predictions of three image classifiers.

However, evaluation in this field has been inconsistent. Most papers use different models, different datasets, and inconsistent metrics. Furthermore, existing tools often focus almost exclusively on LLMs. There hasn't been a single place to compare a weight-merging algorithm against a mixture-of-experts (MoE) approach across diverse architectures like ViTs, ResNets, and Transformers. FusionBench fills this gap by providing a modular codebase where any fusion algorithm can be tested against a standardized "Task Pool."

The FusionBench Framework

FusionBench introduces a modular architecture designed to bring rigorous evaluation to the world of model fusion. The library is built on top of Hydra for configuration management, making experiments reproducible with simple YAML files. It also introduces technical utilities like LazyStateDict to handle large-scale model fusion on consumer-grade GPUs by loading parameters only when needed.

Overview of the FusionBench framework architecture.

The framework consists of three primary components:

  1. Model Pool Module: A unified interface for managing source models, such as a collection of CLIP models fine-tuned on different tasks.
  2. Algorithm Module: Implements a wide array of fusion strategies. These are categorized into three main types:
  3. Model Ensemble: Combining the outputs or predictions of multiple models.
  4. Model Merging: Integrating different models into a single one by combining weights into the same architecture.
  5. Model Mixing: Upscaling or recombining components into a new, often heterogeneous architecture, such as a Mixture-of-Experts (MoE).
  6. Task Pool Module: Standardizes evaluation, ensuring that metrics like "Accuracy" are consistent across all tested algorithms.
Comparison of Model Ensemble, Model Merging, and Model Mixing strategies.

Evaluation Suite and Datasets

The benchmark provides a massive evaluation suite covering both Computer Vision (CV) and Natural Language Processing (NLP):

  • CLIP-ViT: 20 different image classification tasks (e.g., SUN397, EuroSAT).
  • ResNet-50: Multi-task scene understanding using the NYUv2 dataset (segmentation, depth, and surface normals).
  • GPT-2 & Flan-T5: Eight NLP tasks from the GLUE benchmark.
  • Mistral-7B: Scaling tests using MMLU, GSM8K, and ARC Challenge to evaluate fusion at the 7-billion parameter scale.

Key Insights and Results

Experimental results from FusionBench provide several critical insights into the nature of model fusion:

  • Orthogonality of Task Vectors: Researchers found that "task vectors" (the difference between a fine-tuned model and its pre-trained base) are nearly orthogonal. This suggests that knowledge for different tasks lives in distinct "directions" in the weight space, explaining why merging often works without massive interference.
  • Scale and Efficiency: Larger models (like ViT-L vs. ViT-B) are generally easier to fuse effectively, likely due to having more "room" in their parameter space. Using LazyStateDict, the authors fused 7B-parameter models on a single NVIDIA RTX 3090 (24GB).
  • Mixing vs. Merging: While simple weight averaging is a popular baseline, "Model Mixing" (e.g., turning specialized models into an MoE) consistently provides superior performance by preserving specialized expertise.
  • Performance Benchmarks: On CLIP-ViT-L tasks, the best fusion methods (like WEMoE and SMILE) achieved an average accuracy of 93.8%, remarkably close to the 94.3% of individual fine-tuning and exceeding the 92.4% of traditional Multi-Task Learning (MTL).
  • Generalization: Fused models showed significantly better "zero-shot" performance on unseen tasks compared to individual specialized models, indicating improved robustness.

Conclusion

FusionBench provides the first comprehensive, open-source benchmark and modular library dedicated to deep model fusion. While fusion methods are closing the gap to traditional MTL, challenges remain in areas like extreme data robustness and lossless fusion. FusionBench is intended to become the standard sandbox for testing the next generation of model merging algorithms. The code and documentation are available at github.com/tanganke/fusion_bench.

Summary updated

Abstract

Deep model fusion is an emerging technique that unifies the predictions or parameters of several deep neural networks into a single model in a cost-effective and data-efficient manner. This enables the unified model to take advantage of the original models' strengths, potentially exceeding their performance. Although a variety of deep model fusion techniques have been introduced, their evaluations tend to be inconsistent and often inadequate to validate their effectiveness and robustness against distribution shifts. To address this issue, we introduce FusionBench, which is the first comprehensive benchmark dedicated to deep model fusion. FusionBench covers a wide range of tasks, including open-vocabulary image classification, text classification, and text-to-text generation. Each category includes up to eight tasks with corresponding task-specific models, featuring both full fine-tuning and LoRA fine-tuning, as well as models of different sizes, to ensure fair and balanced comparisons of various multi-task model fusion techniques across different tasks, model scales, and fine-tuning strategies. We implement and evaluate a broad spectrum of deep model fusion techniques. These techniques range from model ensemble methods, which combine the predictions to improve the overall performance, to model merging, which integrates different models into a single one, and model mixing methods, which upscale or recombine the components of the original models. FusionBench now contains 26 distinct tasks, 74 fine-tuned models, and 16 fusion techniques, and we are committed to consistently expanding the benchmark with more tasks, models, and fusion techniques. In addition, we offer a well-documented set of resources and guidelines to aid researchers in understanding and replicating the benchmark results. Homepage https://github.com/tanganke/fusion_bench
BibTeX
@misc{tang2024fusionbench,
  author = {Anke Tang and Li Shen and Yong Luo and Han Hu and Bo Du and Dacheng Tao},
  title = {{FusionBench: A Comprehensive Benchmark of Deep Model Fusion}},
  year = {2024},
  eprint = {2406.03280},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2406.03280},
}
Figure