TL;DR
FusionBench is the first comprehensive benchmark and unified open-source library designed specifically for deep model fusion across both computer vision and natural language processing. It provides a standardized, modular framework to implement and evaluate diverse fusion paradigms—ensembling, merging, and mixing—helping researchers systematically understand how to combine multiple specialized models into a single, robust network without training from scratch.
What Problem Does This Paper Address?
Deep model fusion is a highly cost-effective and data-efficient paradigm that unifies the parameters or predictions of multiple deep neural networks into a single, superior model. However, as the field has rapidly evolved, evaluating these techniques has become highly inconsistent. Existing toolkits (such as MergeKit) focus almost exclusively on large language model (LLM) merging within natural language processing and lack standardized evaluation frameworks.
FusionBench addresses this gap by providing a unified platform that spans multiple architectures, model scales, and domains (Computer Vision and NLP), allowing researchers to benchmark the effectiveness, robustness, and generalization capabilities of fusion algorithms under identical conditions.
Methods
FusionBench introduces a modular and flexible architecture built around three primary components:
- Algorithm Module: Contains implementations of cutting-edge fusion algorithms categorized into three main paradigms: Model Ensemble (combining predictions), Model Merging (integrating parameters of isomorphic models), and Model Mixing (fusing components to create new heterogeneous models, like Mixture-of-Experts). It also includes complementary model compression methods.
- Model Pool Module: Provides a unified interface for managing and loading diverse model architectures and fine-tuned weights.
- Task Pool Module: Standardizes evaluation by pairing specific datasets with their corresponding evaluation metrics.
The general framework of the codebase is illustrated below:
The authors also propose a clear taxonomy to categorize the various deep model fusion techniques:
All experimental setups are streamlined via a single unified interface using Hydra-based YAML configurations, making it easy to integrate new algorithms or tasks.
Datasets
The benchmark incorporates a diverse set of tasks and datasets across domains:
- Open-Vocabulary Image Classification: CLIP-ViT (B/32 and L/14) models fine-tuned on up to 20 diverse datasets, including SUN397, Cars, RESISC45, EuroSAT, SVHN, GTSRB, MNIST, and DTD.
- Scene Understanding: ResNet-50 models evaluated on the NYUv2 dataset for semantic segmentation, depth estimation, and surface normal estimation.
- Text Classification & Generation: GPT-2 and LoRA fine-tuned Flan-T5 (Base and Large) models evaluated on GLUE benchmark tasks (including CoLA, MNLI, MRPC, QNLI, QQP, RTE, SST-2, and STSB).
- LLM Scaling: Mistral-7B models evaluated on standard language benchmarks including MMLU, TruthfulQA, GSM8K, and ARC Challenge.
Key Insights and Ideas
- Orthogonality of Task Vectors: The authors discovered that "task vectors" (the difference between a fine-tuned model's parameters and its pre-trained initialization) across different tasks are nearly orthogonal. This suggests that task-specific knowledge resides in distinct directions or subspaces, explaining why subspace-based merging methods can successfully isolate and combine skills.
- Model Mixing Superiority: Model mixing approaches (such as Weight-Ensemble MoE and SMILE) consistently achieve top-tier performance, frequently approaching or matching traditional multi-task learning (MTL) baselines where models are trained on all datasets simultaneously.
- Scale Facilitates Fusion: The performance gap between model fusion techniques and individual fine-tuned models narrows as the model scale increases (e.g., CLIP-ViT-L/14 vs. CLIP-ViT-B/32). This indicates that larger foundation models provide more robust and flexible parameter spaces that are inherently more amenable to fusion.
- The Robustness Bottleneck: While advanced test-time adaptation methods (like AdaMerging) show excellent generalization on clean data, all fusion methods suffer severe performance drops under extreme out-of-distribution data corruptions (such as pixelation or impulse noise).
Key Results
- Image Classification: On the CLIP-ViT-B/32 benchmark, model mixing methods like WEMoE (89.2%) and SMILE (89.3%) significantly outperformed simple weight averaging (66.5%) and closely matched traditional multi-task learning (88.6%).
- Scene Understanding: Multi-task fusion on ResNet-50 successfully balanced performance across segmentation, depth, and normal estimation, mitigating the catastrophic drop in performance on non-target tasks typically seen in single-task experts.
- LLM Scaling: Upscaling Mistral-7B models using the SMILE upscaling algorithm yielded an 11.2B parameter model that achieved strong performance on reasoning tasks like GSM8K (70.66%) and TruthfulQA (54.23%), proving that fusion can serve as a cheap alternative to full model pre-training.
Limitations and Future Work
- LLM Evaluation Maturity: Evaluation methodologies for LLMs and Multimodal LLMs (MLLMs) within the toolkit are still maturing, and the high computational cost of evaluating these large-scale networks remains a practical challenge.
- Overfitting to Corrupted Distributions: Under severe distribution shifts, adaptive fusion methods can sometimes overfit to specific tasks, leading to a drop in overall robustness. The authors note that future work must explore better regularization techniques during test-time adaptation.
- Active Expansion: The repository is actively maintained, with plans to continuously integrate emerging fusion algorithms, expand benchmark coverage to larger foundation models, and enhance community-driven contributions.
Summary updated