Towards Performance Consistency in Multi-Level Model Collaboration

Qi Li, Runpeng Yu, Xinchao Wang

TL;DR

This paper bridges the gap between parameter-level model merging and prediction-level ensembling in multi-task learning. They theoretically show that performance consistency is achievable even under relaxed conditions and introduce "Neural Ligand" (NeuLig), a framework that empirically validates this, achieving near-identical performance between merging and ensembling across various scenarios.

What Problem Does This Paper Address?

Multi-task learning (MTL) aims to improve model generalization and efficiency by training a single model on multiple tasks. Two popular approaches are parameter-level model merging (combining model weights) and prediction-level ensembling (averaging model outputs). While merging offers efficiency, it often underperforms ensembling, which is considered the performance ceiling. Prior work suggested merging could match ensembling, but only under strict conditions: using only two models, ViT architectures, and models fine-tuned from the same checkpoint. This paper questions whether this consistency holds when these restrictions are lifted, particularly in scenarios with multiple collaborating models, which is more common in real-world applications.

Methods

The authors first establish a theoretical link between merging and ensembling. They derive that the performance difference between the two is of second-order smallness if a specific condition on the weighted sum of model parameter offsets is met. This theoretical insight guides the development of their validation framework, Neural Ligand (NeuLig).

NeuLig utilizes a small, single-layer neural network called Portland. Portland takes the outputs of multiple collaborating models for a given data point and generates a Cooperative Vector (CoopVec). This CoopVec is then used to perform both parameter-level merging and prediction-level ensembling in a unified manner. The key innovation lies in Portland's specialized loss function, which has two components:

  1. Boosting Term (Lb): Encourages Portland to perform well in prediction-level ensembling.
  2. Alignment Term (La): Aligns the performance of merging with that of ensembling, directly leveraging the theoretical findings about parameter offsets.

This dual-objective training allows NeuLig to guide the CoopVec generation towards achieving performance consistency.

Datasets

The experiments utilize CLIP-like models (CLIP-RN50 and CLIP-ViT-B/32) fine-tuned on five diverse datasets:

  • GTSRB
  • CIFAR10
  • RESISC45
  • CIFAR100
  • MNIST

The paper also explores scenarios with models that are not fine-tuned from the same pre-trained checkpoint, effectively simulating "diverse-origin models" trained from scratch.

Key Insights and Ideas

  • Theoretical Foundation for Consistency: The paper provides a theoretical basis for achieving performance consistency between merging and ensembling, showing that the performance gap can be reduced to second-order smallness under specific conditions related to parameter offsets. This moves beyond empirical observations to a more principled understanding.
  • Unified Framework for Validation: NeuLig offers a novel way to validate merging and ensembling simultaneously. By using Portland to generate CoopVecs, the framework allows for direct comparison and alignment of both paradigms.
  • Resilience to Relaxed Conditions: The key takeaway is that performance consistency is not limited to ideal scenarios. NeuLig demonstrates that merging can indeed match ensembling even when models are from different origins (not fine-tuned from the same checkpoint) and when collaborating in larger groups.
  • ViT vs. ResNet Behavior: The paper highlights interesting differences in how ViT and ResNet architectures behave during merging. ViTs, with their attention mechanisms, seem more amenable to merging due to their more generalized representations, while ResNets' convolutional structures can lead to more interference.

Key Results

  • Near-Perfect Consistency: In a key experiment with 5 CLIP-ViT-B/32 models, parameter-level merging achieved performance (95.44%) virtually identical to prediction-level ensembling (95.46%), with a negligible gap of 0.02%.
  • Robustness: NeuLig demonstrated robust performance consistency across different model scales and numbers of collaborating models (up to 5).
  • Effectiveness on Diverse-Origin Models: The framework successfully achieved performance consistency even when models were randomly initialized and trained from scratch, a scenario where existing methods failed.
  • Efficiency: NeuLig showed competitive or superior performance compared to baseline merging methods while maintaining reasonable resource usage.

Limitations and Future Work

  • Computational Cost: While NeuLig is a validation framework, the training of Portland itself adds a computational overhead. The paper does not extensively detail the computational cost of NeuLig compared to standard merging methods in all scenarios.
  • Generalizability to Other Architectures: The primary focus is on CLIP-like models (ViT and ResNet). Further exploration of NeuLig's effectiveness on a wider range of architectures could be beneficial.
  • Semi-Supervised vs. Supervised: The paper presents both supervised and semi-supervised setups for Portland's training. While the semi-supervised approach is more practical when original training data is unavailable, further analysis on the trade-offs and performance differences between these two could be valuable.
  • Future Work: The authors suggest exploring the CoopVec Map for dataset-level merging/ensembling and investigating the impact of varying data scales on performance consistency, which they begin to address in the paper. Further research into the theoretical underpinnings and practical applications of NeuLig in more complex multi-task learning scenarios is also implied.

Summary updated

Abstract

Parameter-level model merging is an emerging paradigm in multi-task learning with significant promise. Previous research has explored its connections with prediction-level model ensembling-commonly viewed as the upper bound for merging-to reveal the potential of achieving performance consistency between the two. However, this observation relies on certain preconditions, such as being limited to two models, using ViT-based models, and all models are fine-tuned from the same pre-trained checkpoint. To further understand the intrinsic connections between model merging and model ensembling, this paper explores an interesting possibility: If these restrictions are removed, can performance consistency still be achieved between merging and ensembling? To answer this question, we first theoretically establish a performance correlation between merging and ensembling. We find that even when previous restrictions are not met, there is still a way for model merging to attain a near-identical and superior performance similar to that of ensembling. To verify whether our findings are practical, we introduce a validation framework termed Neural Ligand (NeuLig). The learning process of NeuLig is meticulously designed with a specialized loss function supported by theoretical foundations. Experimental results demonstrate the robust resilience of NeuLig in terms of both model scale and the number of collaborating models. For instance, for the case involving 5 CLIP-ViT-B/32 models, parameter-level merging achieves the same performance as prediction-level ensembling (merging: 95.44% vs. ensembling: 95.46%).
BibTeX
@inproceedings{li2025towards,
  author = {Qi Li and Runpeng Yu and Xinchao Wang},
  title = {{Towards Performance Consistency in Multi-Level Model Collaboration}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2025},
  url = {https://openreview.net/forum?id=wojB1enRLZ},
}
Figure