Model Reuse With Reduced Kernel Mean Embedding Specification

Xi-Zhu Wu, Wenkai Xu, Song Liu, Zhi-Hua ZhouIEEE TKDE 2020 · OpenReview

TL;DR

This paper introduces a novel "learnware" framework that allows users to reuse pre-trained machine learning models without accessing their original training data. They achieve this by creating a compact "Reduced Kernel Mean Embedding" (RKME) specification for each model, enabling efficient model selection for new tasks. This approach addresses the critical challenge of model reuse while preserving data privacy.

What Problem Does This Paper Address?

Imagine a vast online marketplace filled with pre-trained machine learning models, each trained for a specific task. Now, you have a new, related task and want to leverage these existing models instead of starting from scratch. The core problem is: how do you find the right pre-trained models from this pool that will be genuinely helpful for your new task, especially when you can't see the data they were trained on? This is crucial for efficiency and also for privacy, as raw training data often contains sensitive information. Existing methods struggle with this, either assuming all models are useful or requiring access to data.

Methods

The authors propose a two-phase framework:

  • Upload Phase: When a model provider uploads their pre-trained model, they don't upload the raw data. Instead, they compute a Reduced Kernel Mean Embedding (RKME). This RKME acts as a compact, privacy-preserving "specification" of the model's training data distribution. It's essentially a summary of the data's characteristics in a high-dimensional feature space (RKHS), but without revealing the individual data points. They use techniques like reduced set construction to create this compact representation.
  • Deployment Phase: When a user has a new task and unlabeled test data, they use the RKME specifications from the pool. The framework considers two main assumptions about the relationship between the new task and the existing tasks:
  • Task-recurrent: The new task's data distribution is identical to one of the existing tasks. They find the best match by minimizing the RKHS distance between the test data's empirical KME and the stored RKMES.
  • Instance-recurrent: The new task's data distribution is a mixture of existing tasks' distributions. They estimate the mixing weights, generate synthetic data that mimics this mixture using kernel herding, and then train a simple "selector" model to decide which pre-trained model to use for each individual test data point.

Datasets

The paper evaluates their approach on:

  • Toy Example: A synthetic binary classification problem with three providers, demonstrating the core concepts visually.
  • Benchmark Datasets:
    • CIFAR-100: Used to simulate a setting with 20 local datasets (each representing a superclass) and 5-class local neural network classifiers.
  • 20-newsgroup: Used similarly, with 5 superclasses as local datasets and local models.
  • Real-World Project: A project at Huawei Technologies Co., Ltd. involving predicting communication quality gains from "deep cover" technology on base stations. This involved 8 anonymous base stations, treated as solved tasks.

Key Insights and Ideas

  • RKME as a Privacy-Preserving Specification: The core innovation is using RKME to summarize data distributions without revealing raw data. This is a powerful concept for enabling model reuse in privacy-sensitive scenarios.
  • Two Realistic Assumptions for Reuse: The task-recurrent and instance-recurrent assumptions provide a flexible framework for understanding how new tasks relate to existing ones, covering both direct matches and mixtures.
  • Kernel Herding for Mimicry: Using kernel herding to generate synthetic data for the instance-recurrent case is a clever way to create a representative dataset for training the selector model, without needing access to real data.
  • Dynamic Model Selection: The instance-recurrent approach allows for dynamic selection of the best pre-trained model for each test instance, making it adaptive and robust to complex data distributions.

Key Results

  • Task-Recurrent: The proposed method achieves significantly higher accuracy than baselines (MAX, HMR) on CIFAR-100 and 20-newsgroup datasets when the task-recurrent assumption holds, often outperforming even a global model trained on merged data. For instance, on CIFAR-100 with 1 mixing task, their method achieved 86.22% accuracy compared to 70.58% for HMR.
  • Instance-Recurrent: The method performs competitively, often outperforming baselines and sometimes even the global model, especially when the number of mixing tasks is smaller. For example, on 20-newsgroup with 5 mixing tasks, their method achieved 72.68% accuracy, while the global model achieved 73.24%.
  • Real-World Project (Regression): In a regression task predicting communication quality gain, their method achieved a lower RMSE (0.0279) compared to baselines like RAND (0.0363) and AVG (0.0326), and even outperformed transfer learning methods like mSDA (0.0285) in some metrics, all while maintaining privacy.

Limitations and Future Work

  • Assumption Dependency: The performance is tied to the validity of the task-recurrent or instance-recurrent assumptions. If the new task's distribution is vastly different and not a mixture, performance might degrade.
  • Kernel Choice: The effectiveness of RKME depends on the choice of kernel function.
  • Future Work: The authors plan to explore more powerful kernel methods for constructing RKMEs and investigate even weaker assumptions for model reuse. They also aim to design other types of valid specifications.

Summary updated

Abstract

Given a publicly available pool of machine learning models constructed for various tasks, when a user plans to build a model for her own machine learning application, is it possible to build upon models in the pool such that the previous efforts on these existing models can be reused rather than starting from scratch? Here, a grand challenge is how to find models that are helpful for the current application, without accessing the raw training data for the models in the pool. In this paper, we present a two-phase framework. In the upload phase, when a model is uploading into the pool, we construct a reduced kernel mean embedding (RKME) as a <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">specification</i> for the model. Then in the deployment phase, the relatedness of the current task and pre-trained models will be measured based on the value of the RKME specification. Theoretical results and extensive experiments validate the effectiveness of our approach.
BibTeX
@article{wu2020model,
  author = {Xi-Zhu Wu and Wenkai Xu and Song Liu and Zhi-Hua Zhou},
  title = {{Model Reuse With Reduced Kernel Mean Embedding Specification}},
  journal = {IEEE Trans. Knowl. Data Eng.},
  year = {2020},
  url = {https://openreview.net/forum?id=RAsOiBqLGSV},
}
Figure