Optimize Incompatible Parameters Through Compatibility-aware Knowledge Integration

Zheqi Lv, Keming Ye, Zishu Wei, Qi Tian, Shengyu Zhang, Wenqiao Zhang, Wenjie Wang, Kun Kuang, Tat-Seng Chua, Fei Wu

TL;DR

This paper introduces Compatibility-aware Knowledge Integration (CKI), a novel method to improve deep learning models by intelligently integrating knowledge from multiple pre-trained models. CKI identifies and leverages "incompatible parameters" – those that hinder performance on specific data distributions – to enhance model robustness and accuracy without increasing inference costs.

What Problem Does This Paper Address?

Deep neural networks, despite their successes, often contain "incompatible parameters." These are parameters that, while potentially useful in some contexts, can negatively impact a model's performance when faced with specific or shifting data distributions. Existing solutions either prune these parameters (sacrificing potential performance for efficiency) or ensemble model outputs (increasing computational cost significantly during inference). This paper tackles the challenge of directly improving these incompatible parameters by integrating knowledge from diverse models, aiming for enhanced performance without the overhead of traditional ensemble methods.

Methods

The core of the proposed solution is the Compatibility-aware Knowledge Integration (CKI) framework, which consists of two main components:

  • Parameter Compatibility Assessment: This module evaluates how compatible parameters are across different models. It employs a dual-perspective approach:
  • Local-level Parameter Uncertainty: Measures the variance in parameter values at analogous positions across multiple model instances. High variance suggests uncertainty and potential incompatibility.
  • Global-level Model Information Content: Assesses the overall richness and diversity of information within a model by analyzing the entropy of its parameter distribution. Models with higher entropy are considered to contain more significant information.
  • These two aspects are fused to create a comprehensive "dual-perspective parameter compatibility" metric.
  • Parameter Splicing: Based on the compatibility assessment, this module integrates knowledge from multiple models into a single, enhanced model. Two splicing methods are proposed:
  • Hard Splicing: Directly replaces parameters with lower compatibility with their counterparts from models exhibiting higher compatibility.
  • Soft Splicing: A more nuanced approach that weights and fuses parameters based on their compatibility scores, aiming to preserve more information.

Crucially, the integrated model retains the same structure and parameter count as the original pre-trained models, ensuring no additional inference cost.

Datasets

The paper evaluates CKI on several widely-used datasets:

  • Recommendation Tasks: Amazon-Beauty, Douban-Book, Douban-Music, and Movielens-1M. These datasets are noted for their pronounced and rapid data distribution shifts.
  • Language Tasks: RTE and SST-2.

Key Insights and Ideas

  • "Incompatible Parameters" are a real issue: The paper highlights that not all parameters in a trained model are equally beneficial across all data distributions, and some can even be detrimental.
  • Beyond Pruning and Ensembling: CKI offers a novel third way to address incompatible parameters, focusing on integration and enhancement rather than removal or costly output combination.
  • Dual-Perspective Compatibility is Key: Combining local parameter uncertainty with global model information content provides a more robust measure of parameter compatibility than either metric alone.
  • Soft Splicing Outperforms Hard Splicing: The experimental results suggest that a weighted fusion of parameters (soft splicing) is more effective than a direct replacement (hard splicing), likely due to better preservation of valuable information.
  • CKI as a Better Initialization: The integrated model produced by CKI can serve as an excellent starting point for further training, achieving convergence in just one epoch.

Key Results

  • CKI consistently outperforms baseline methods (Model Pruning, Output Ensemble, Parameter Averaging) across various recommendation and language tasks.
  • On recommendation tasks, CKI achieved significant improvements in metrics like NDCG@5 and HR@5, often surpassing the best pre-trained models. For example, on the Douban Music dataset, CKI improved NDCG@5 from 0.3718 to 0.3756 when using GRU4Rec.
  • The integrated model has the same inference cost as the original models, offering a significant advantage over output ensemble methods.
  • Ablation studies confirmed the effectiveness of both the dual-perspective compatibility assessment and the soft splicing method.
  • The impact of the number of integrated models was explored, with optimal performance observed around 7 models, suggesting a trade-off between knowledge integration and potential overfitting or data limitations.

Limitations and Future Work

  • Optimal Number of Models: The paper notes that while integrating more models generally improves performance, there's an optimal point (around 7 models in their experiments). Exceeding this can lead to performance degradation, possibly due to increased network size and data limitations. This suggests future work could explore more sophisticated methods for selecting or weighting models when a large number are available.
  • Generalization Theory: While the paper mentions exploring the generalization boundary of the spliced model, a deeper theoretical understanding of CKI's generalization capabilities could be a future research direction.
  • Dynamic Model Integration: The paper shows promising results when fusing static and dynamic models, suggesting further exploration into optimizing dynamic recommendation systems with CKI.

Summary updated

BibTeX
@inproceedings{lv2025optimize,
  author = {Zheqi Lv and Keming Ye and Zishu Wei and Qi Tian and Shengyu Zhang and Wenqiao Zhang and Wenjie Wang and Kun Kuang and Tat-Seng Chua and Fei Wu},
  title = {{Optimize Incompatible Parameters Through Compatibility-aware Knowledge Integration}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i18.34117},
}
Figure