TL;DR
FusionBench is the first comprehensive, modular benchmark designed to systematically evaluate deep model fusion techniques across computer vision and NLP. Spanning 74 fine-tuned models, 19 fusion algorithms, and diverse evaluation settings, it establishes a unified taxonomy (ensembling, merging, and mixing) and reveals key trade-offs in multi-task transfer, robustness against distribution shifts, and generalization to unseen tasks.
What Problem Does This Paper Address?
Deep model fusion—combining the parameters or predictions of multiple task-specific models into a unified architecture—has emerged as an efficient alternative to training multi-task foundation models from scratch. However, the sub-field lacks standardized evaluation benchmarks. Existing studies typically evaluate new algorithms on bespoke, inconsistent testbeds, use disparate fine-tuning configurations (full weights vs. parameter-efficient LoRA), and rarely test fused models against distribution shifts or unseen tasks. FusionBench fills this void with a modular platform to evaluate fusion algorithms under standardized, reproducible conditions.
Methods
The authors structure model fusion into a three-part taxonomy:
- Model Ensemble: Combining model predictions at inference (e.g., Simple/Weighted Ensembles, Max-Model Predictor). Effective, but carries high inference and memory overhead.
- Model Merging: Combining parameters of isomorphic architectures into a single model of identical size without extra inference cost (e.g., Weight Averaging / Model Soups, Fisher Merging, RegMean, Task Arithmetic, TIES-Merging, and test-time adaptive methods like AdaMerging).
- Model Mixing: Recombining or upscaling components into a new, often heterogeneous or sparse architecture (e.g., Sparse MoE upcycling, SMILE, Weight-Ensemble MoE [WEMoE], and depth upscaling).
The open-source framework is decoupled into three primary configurable modules managed via a YAML-driven CLI:
- Algorithm Module: Encapsulates fusion algorithms as self-contained Python classes.
- Model Pool Module: Standardizes checkpoint loading, metadata tagging, and adaptation hooks.
- Task Pool Module: Manages evaluation datasets, corruptions, and task-specific metrics.
Datasets
FusionBench evaluates cross-modal capabilities across several standard benchmark suites:
- Open-Vocabulary Image Classification: Evaluated on 8 datasets (SUN397, Stanford Cars, RESISC45, EuroSAT, SVHN, GTSRB, MNIST, DTD) using CLIP-ViT-B/32, CLIP-ViT-B/16, and CLIP-ViT-L/14 backbones. Includes 7 corrupted test distributions (motion blur, impulse/Gaussian noise, pixelation, spatter, contrast, JPEG compression) based on ImageNet-C protocols.
- Scene Understanding: 3 dense-prediction tasks (13-class segmentation, depth estimation, and surface normal estimation) on the indoor NYUv2 dataset with ResNet-50.
- Text Classification & Text-to-Text Generation: GLUE tasks (CoLA, MNLI, MRPC, QNLI, QQP, RTE, SST-2, STSB) utilizing GPT-2, Flan-T5-Base, and Flan-T5-Large (with and without LoRA).
- LLM Reasoning & Knowledge: Fusing 7B models (Mistral-7B derivatives) evaluated on MMLU, GSM8K, TruthfulQA, and ARC Challenge.
Key Insights and Ideas
- Task Vectors are Nearly Orthogonal: Analyzing cosine similarity across fine-tuned CLIP task vectors revealed values close to zero (often $\le 0.05$). This empirical near-orthogonality explains why linear parameter addition (e.g., Task Arithmetic) works, but also shows that interference happens in subtle, shared subspaces.
- Adaptive Merging and MoE Mixing Outperform Static Averages: Methods dynamically routing weights or adapting parameters at test time (AdaMerging, WEMoE, SMILE) consistently beat static weight averaging. Dynamic routing preserves distinct task capabilities that parameter averaging washes out.
- Smaller Models Gain More from Fusion: Fusing CLIP-ViT-B/32 models showed a wider performance gap relative to single-task learning (STL) than CLIP-ViT-L/14 models. Larger pre-trained representations already capture broad cross-task features, leaving less room for linear multi-task gains.
- Severe Vulnerability to Corruptions and Negative Transfer: When tested on unseen tasks or severe image corruptions (like impulse noise or pixelation), merged models can suffer severe performance drops—sometimes performing worse than the original zero-shot pre-trained base model (negative transfer). Test-time adaptive methods also risk overfitting to corrupted evaluation inputs.
Key Results
- CLIP-ViT-B/32 8-Task Merging: Individual fine-tuned models averaged 90.3% accuracy, while the pre-trained base achieved 48.2%. Simple weight averaging reached only 66.5%. Advanced merging methods like Layer-Wise AdaMerging reached 82.6%, and model mixing methods reached 89.3% (SMILE) and 89.2% (WEMoE), approaching the traditional multi-task learning (MTL) upper bound (88.6%).
- Flan-T5-Base Multi-Task Merging: Starting from a pre-trained average of 75.7% across 8 GLUE tasks, simple averaging achieved 78.2%, while SMILE model mixing matched individual LoRA fine-tuning performance closely (84.0% vs. 84.6% individual average).
- LLM Upscaling (Mistral-7B): Merging MetaMath, Dolphin-2.1, and Speechless-Code via SMILE dense-expert upscaling (11.2B total parameters) achieved 70.66% on GSM8K and 54.23% on TruthfulQA, outperforming the individual source models on balanced multi-discipline benchmarks without additional training.
Limitations and Future Work
- Primary Focus on Multi-Task Fusion: While the codebase implements ensembling and mixing algorithms intended for wider use cases, the current systematic benchmark focuses almost exclusively on multi-task model fusion.
- Evaluation Cost for Frontier LLMs: Due to the high computational barrier of LLM evaluations, large language model benchmark suites are currently evaluated via external harnesses rather than natively inside the core benchmark pipelines.
- Future Directions: The authors plan to extend FusionBench to RLHF/human preference alignment fusion, multimodal foundation model combinations, and continuous reinforcement learning scenarios.
Summary updated