Introduction
Deep learning's computational demands have spurred significant interest in model merging and pruning. Model merging aims to combine multiple pre-trained models into a single entity to leverage diverse learned features, while pruning seeks to reduce model size by removing redundant weights. However, merging models with different initializations often leads to a catastrophic drop in performance, frequently resulting in accuracy no better than random chance. This phenomenon, often termed "variance collapse," has been addressed with heuristic patches like the REPAIR algorithm, but a fundamental understanding of its cause has remained elusive.
This paper identifies the root cause as the vanishing feature phenomenon. We demonstrate that as signals travel through a merged or pruned network, the input-induced features exponentially decay, leaving the output dominated by input-independent biases. To mitigate this, we propose Preserve-First Merging (PFM), a strategy that maintains separate early layers to preserve signal quality, enabling merged models to outperform their parents without retraining.
The Vanishing Feature Phenomenon
To analyze this effect, we decompose the activation of any layer into two components:
- $g(x)$ (Input-induced features): The signal derived directly from the input data.
- $h$ (Input-independent offset): The noise or bias terms that remain constant regardless of the input.
In linear networks, merging models by averaging weights acts as a scaling factor. If the models are not perfectly aligned, this scaling factor is less than one. Because deep networks involve successive multiplications across layers, the signal $g(x)$ decays exponentially. By the final layers, the signal has effectively vanished, and the output is dominated by the accumulated bias $h$.
This decay is not limited to linear models. As shown in our analysis of general networks, the signal-to-noise ratio drops significantly as depth increases, particularly in architectures without residual connections.
Preserve-First Merging (PFM)
Our discovery suggests that the quality of the signal in the early layers is critical. PFM addresses this by keeping the first $l$ layers of each model separate in a "multi-start" architecture. Inputs are processed independently through these preserved layers, and only the subsequent deeper layers are merged. This ensures that a high-quality signal is established before it enters the potentially lossy merged sections of the network.
Experimental Evaluation
We validated the vanishing feature phenomenon and the efficacy of PFM across standard benchmarks including MNIST, CIFAR-10/100, and ImageNet, using architectures such as VGG, ResNet, ConvNeXt, and Vision Transformers (ViT).
Key Findings:
- Performance Gains: Using PFM, merged VGG16 models on CIFAR-10 outperformed their individual parent models without any fine-tuning—a first in the field of model merging.
- Architecture Resilience: ResNets are naturally more "mergeable" than VGG-style networks because residual connections act as shortcuts that preserve early-layer features, preventing rapid signal decay (see Figure 5B).
- Pruning Improvements: The vanishing feature lens also explains why one-shot pruning fails at high sparsity. By applying normalization insights (REPAIR/RESET) to account for magnitude reduction, we improved the accuracy of pruned ConvNeXt-B models on ImageNet by 36-55% at 80% sparsity.
Conclusion
The vanishing feature phenomenon provides a unifying explanation for the failures of model merging and pruning. By decomposing activations into signal and bias, we can better understand how architectural choices like residual connections and normalization layers impact model connectivity. Preserve-First Merging offers a practical, training-free path toward efficient model ensemble and compression, though future work is needed to optimize the computational overhead of its multi-start design.
Summary updated