Optimize Incompatible Parameters Through Compatibility-aware Knowledge Integration

Zheqi Lv, Keming Ye, Zishu Wei, Qi Tian, Shengyu Zhang, Wenqiao Zhang, Wenjie Wang, Kun Kuang, Tat-Seng Chua, Fei Wu

Abstract

Deep neural networks often contain "incompatible parameters"—weights that are redundant or detrimental to performance. While pruning and output ensembling offer partial solutions, they suffer from significant trade-offs in accuracy and computational efficiency. This paper introduces Compatibility-aware Knowledge Integration (CKI), a framework that identifies and replaces these incompatible parameters by splicing together the most effective components from multiple pre-trained models. CKI achieves the performance gains typically associated with large ensembles while maintaining the inference speed and memory footprint of a single model.

Introduction

Despite the success of deep neural networks, research into the "lottery ticket hypothesis" suggests that only a small subnetwork is often responsible for a model's success. The remaining weights, termed incompatible parameters, can be redundant or even harmful when the model encounters specific data distributions.

Existing methods to address this inefficiency generally fall into two categories:

  1. Pruning: Removing parameters to boost efficiency, which often sacrifices raw performance in dynamic environments.
  2. Output Ensembling: Averaging predictions from multiple models, which yields high accuracy but multiplies computational and storage costs during inference.

CKI proposes a middle ground: integrating the complementary strengths of different models into a single, optimized architecture without adding any inference overhead.

The CKI Framework

Compatibility-aware Knowledge Integration (CKI) treats model parameters like interchangeable parts. The process consists of two primary stages: assessment and splicing.

1. Parameter Compatibility Assessment

To determine which parameters are "compatible" (beneficial) versus "incompatible" (detrimental), CKI evaluates models through two distinct lenses:

  • Local-level Uncertainty: By comparing analogous parameters across different model instances (e.g., models trained with different random seeds), the framework identifies "uncertain" parameters. High variance across models suggests a parameter position is unreliable.
  • Global-level Information Content: The framework calculates the information entropy of a model’s entire parameter distribution. Higher entropy is used as a proxy for a richer, more robust set of features and higher "knowledge diversity."

These metrics are fused into a single compatibility score for every parameter position in the network.

2. Parameter Splicing

Once the compatibility scores are established, CKI merges the models using one of two strategies:

  • Hard Splicing: A "winner-takes-all" approach where the most compatible parameter for each specific position is selected from the pool of models.
  • Soft Splicing: A weighted averaging approach where parameters are fused based on their compatibility scores, minimizing potential information loss from outright rejection.

The resulting "spliced" model maintains the exact same structure as the original pre-trained models, allowing for immediate deployment with zero additional latency.

Experimental Evaluation

The researchers evaluated CKI across two distinct domains to demonstrate its universality: Recommender Systems and Natural Language Processing (NLP).

Recommender Systems

Tested on datasets including Amazon-Beauty, Douban-Book, Douban-Music, and Movielens-1M, CKI addressed the challenge of rapid shifts in user preferences. On the Amazon-Beauty dataset, CKI achieved an NDCG@5 of 0.1409, significantly outperforming the base Model A (0.1137) and even the computationally expensive Output Ensemble (0.1193).

Natural Language Processing

Using the GLUE benchmark (RTE and SST-2 datasets), CKI demonstrated similar gains. On the SST-2 sentiment task, CKI reached an accuracy of 0.8544, outperforming both the best individual model (0.8440) and the ensemble (0.8532).

Efficiency Metrics

Crucially, CKI achieved these performance gains with a Cost of 1.0 (1x inference time). In contrast, the Output Ensemble required a cost of 2.0 or more to achieve comparable or lower results, proving CKI's efficiency in resource-constrained environments.

Key Insights

  • Replacement over Removal: CKI shifts the paradigm from removing bad parameters (pruning) to replacing them with superior versions from "peer" models.
  • Superior Initialization: A CKI-integrated model serves as an excellent starting point for further refinement. Re-training a spliced model for just one epoch led to even higher performance, suggesting CKI identifies a more optimal location in the loss landscape than standard random initialization.

Conclusion and Future Work

CKI provides a robust framework for "repairing" deep neural networks by intelligently integrating knowledge across models. While the framework currently requires an upfront cost to train multiple initial models, the long-term inference benefits are substantial.

Future research may explore more sophisticated "Soft Splicing" techniques and investigate how CKI scales to Large Language Models (LLMs), where the sheer volume of parameters makes compatibility assessment more computationally demanding. Additionally, extending this approach to computer vision and audio processing remains a promising avenue for exploration.

Summary updated

Abstract

Deep neural networks have become foundational to advancements in multiple domains, including recommendation systems, natural language processing, and so on. Despite their successes, these models often contain incompatible parameters that can be underutilized or detrimental to model performance, particularly when faced with specific, varying data distributions. Existing research excels in removing such parameters or merging the outputs of multiple different pretrained models. However, the former focuses on efficiency rather than performance, while the latter requires several times more computing and storage resources to support inference. In this paper, we set the goal to explicitly improve these incompatible parameters by leveraging the complementary strengths of different models, thereby directly enhancing the models without any additional parameters. Specifically, we propose Compatibility-aware Knowledge Integration (CKI), which consists of Parameter Compatibility Assessment and Parameter Splicing, which are used to evaluate the knowledge content of multiple models and integrate the knowledge into one model, respectively. The integrated model can be used directly for inference or for further fine-tuning. We conduct extensive experiments on various datasets for recommendation and language tasks, and the results show that Compatibility-aware Knowledge Integration can effectively optimize incompatible parameters under multiple tasks and settings to break through the training limit of the original model without increasing the inference cost.
BibTeX
@inproceedings{lv2025optimize,
  author = {Zheqi Lv and Keming Ye and Zishu Wei and Qi Tian and Shengyu Zhang and Wenqiao Zhang and Wenjie Wang and Kun Kuang and Tat-Seng Chua and Fei Wu},
  title = {{Optimize Incompatible Parameters Through Compatibility-aware Knowledge Integration}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i18.34117},
}
Figure