Abstract
Deep neural networks often contain "incompatible parameters"—weights that are redundant or detrimental to performance. While pruning and output ensembling offer partial solutions, they suffer from significant trade-offs in accuracy and computational efficiency. This paper introduces Compatibility-aware Knowledge Integration (CKI), a framework that identifies and replaces these incompatible parameters by splicing together the most effective components from multiple pre-trained models. CKI achieves the performance gains typically associated with large ensembles while maintaining the inference speed and memory footprint of a single model.
Introduction
Despite the success of deep neural networks, research into the "lottery ticket hypothesis" suggests that only a small subnetwork is often responsible for a model's success. The remaining weights, termed incompatible parameters, can be redundant or even harmful when the model encounters specific data distributions.
Existing methods to address this inefficiency generally fall into two categories:
- Pruning: Removing parameters to boost efficiency, which often sacrifices raw performance in dynamic environments.
- Output Ensembling: Averaging predictions from multiple models, which yields high accuracy but multiplies computational and storage costs during inference.
CKI proposes a middle ground: integrating the complementary strengths of different models into a single, optimized architecture without adding any inference overhead.
The CKI Framework
Compatibility-aware Knowledge Integration (CKI) treats model parameters like interchangeable parts. The process consists of two primary stages: assessment and splicing.
1. Parameter Compatibility Assessment
To determine which parameters are "compatible" (beneficial) versus "incompatible" (detrimental), CKI evaluates models through two distinct lenses:
- Local-level Uncertainty: By comparing analogous parameters across different model instances (e.g., models trained with different random seeds), the framework identifies "uncertain" parameters. High variance across models suggests a parameter position is unreliable.
- Global-level Information Content: The framework calculates the information entropy of a model’s entire parameter distribution. Higher entropy is used as a proxy for a richer, more robust set of features and higher "knowledge diversity."
These metrics are fused into a single compatibility score for every parameter position in the network.
2. Parameter Splicing
Once the compatibility scores are established, CKI merges the models using one of two strategies:
- Hard Splicing: A "winner-takes-all" approach where the most compatible parameter for each specific position is selected from the pool of models.
- Soft Splicing: A weighted averaging approach where parameters are fused based on their compatibility scores, minimizing potential information loss from outright rejection.
The resulting "spliced" model maintains the exact same structure as the original pre-trained models, allowing for immediate deployment with zero additional latency.
Experimental Evaluation
The researchers evaluated CKI across two distinct domains to demonstrate its universality: Recommender Systems and Natural Language Processing (NLP).
Recommender Systems
Tested on datasets including Amazon-Beauty, Douban-Book, Douban-Music, and Movielens-1M, CKI addressed the challenge of rapid shifts in user preferences. On the Amazon-Beauty dataset, CKI achieved an NDCG@5 of 0.1409, significantly outperforming the base Model A (0.1137) and even the computationally expensive Output Ensemble (0.1193).
Natural Language Processing
Using the GLUE benchmark (RTE and SST-2 datasets), CKI demonstrated similar gains. On the SST-2 sentiment task, CKI reached an accuracy of 0.8544, outperforming both the best individual model (0.8440) and the ensemble (0.8532).
Efficiency Metrics
Crucially, CKI achieved these performance gains with a Cost of 1.0 (1x inference time). In contrast, the Output Ensemble required a cost of 2.0 or more to achieve comparable or lower results, proving CKI's efficiency in resource-constrained environments.
Key Insights
- Replacement over Removal: CKI shifts the paradigm from removing bad parameters (pruning) to replacing them with superior versions from "peer" models.
- Superior Initialization: A CKI-integrated model serves as an excellent starting point for further refinement. Re-training a spliced model for just one epoch led to even higher performance, suggesting CKI identifies a more optimal location in the loss landscape than standard random initialization.
Conclusion and Future Work
CKI provides a robust framework for "repairing" deep neural networks by intelligently integrating knowledge across models. While the framework currently requires an upfront cost to train multiple initial models, the long-term inference benefits are substantial.
Future research may explore more sophisticated "Soft Splicing" techniques and investigate how CKI scales to Large Language Models (LLMs), where the sheer volume of parameters makes compatibility assessment more computationally demanding. Additionally, extending this approach to computer vision and audio processing remains a promising avenue for exploration.
Summary updated