Abstract
Model merging offers an efficient way to combine pre-trained neural networks but often suffers from inconsistent performance, especially when merging models with different initializations. We identify the "vanishing feature" phenomenon, where input-induced features diminish during propagation through the merged model, degrading performance. Through theoretical and empirical analysis, we reveal that this phenomenon underpins challenges like variance collapse and explains techniques like permutation-based merging, post-merging normalization, etc. We show that existing normalization strategies can be enhanced by precisely targeting the vanishing feature issue. Leveraging these insights, we propose the "Preserve-First Merging" (PFM) strategy, which focuses on preserving early-layer features, enabling the merged models, for the first time, to outperform the original models in advanced settings without post-training. Furthermore, we demonstrate that the vanishing feature phenomenon extends to other contexts, such as model pruning. Applying post-pruning normalization to mitigate the issue significantly improves one-shot pruning performance at high sparsity, offering a simple and effective post-pruning solution.
Introduction
Model merging is a promising paradigm for efficiently combining multiple pre-trained models into a single architecture without the prohibitive costs of retraining. However, a significant barrier exists: while merging models fine-tuned from a common ancestor is often successful, merging models from different initializations typically results in a "zombie" model with near-zero performance. This paper investigates the root cause of this failure, identifying a phenomenon termed the Vanishing Feature.
The Vanishing Feature Phenomenon
The authors decompose the internal activations of a neural network into two components: the input-induced signal ($g(x)$) and the input-independent bias ($h$). In merged models, the signal $g(x)$ is found to diminish exponentially as it propagates through the network's layers. Simultaneously, the bias component $h$ often grows, eventually overwhelming the signal. This leads to the "variance collapse" observed in prior literature, where the network output becomes insensitive to the input.
Proposed Solutions: PFM and Enhanced Normalization
To counteract the vanishing feature, the paper introduces two primary strategies:
- Preserve-First Merging (PFM): Instead of merging all layers, PFM keeps the initial layers of the parent models separate. By processing inputs through these distinct "heads," the network preserves the signal's strength in the early stages, providing a robust foundation before the features are merged in deeper layers.
- Enhanced Normalization (RESCALE & Bias Calibration): The authors refine existing activation normalization techniques to specifically target the vanishing signal. By amplifying $g(x)$ and suppressing $h$, these methods restore the signal-to-noise ratio within the merged model.
Experimental Evaluation and Key Results
The effectiveness of these strategies was validated across MNIST, CIFAR, and ImageNet benchmarks. A key finding is that architectural choices, such as residual connections, significantly mitigate the vanishing feature effect by providing bypass routes for signal components.
Breaking Performance Ceilings
For the first time, merged VGG16 models on CIFAR-10 achieved performance that surpassed the original parent models without any post-training, thanks to the PFM strategy. This was achieved with only a modest ~20% increase in parameters compared to a standard merge.
Applications in Model Pruning
The vanishing feature insight extends to model pruning, which can be viewed as merging a model with a zero-valued counterpart. Applying the proposed normalization techniques to one-shot pruned models (e.g., Wanda or Magnitude pruning) on ImageNet led to dramatic accuracy recoveries—improving performance by 36% to 55% at 80% sparsity.
Conclusion
By diagnosing the vanishing feature phenomenon, this research provides a unified framework for understanding model merging and pruning failures. The proposed PFM strategy and refined normalization techniques offer computationally efficient paths to creating high-performance merged models, paving the way for more flexible and scalable model composition in the future.
Summary updated