Privacy-preserving Transfer Learning for Knowledge Sharing

Xiawei Guo, Quanming Yao, Wei-Wei Tu, Yuqiang Chen, Wenyuan Dai, Qiang Yang

Abstract

To meet the standard of differential privacy, noise is usually added into the original data, which inevitably deteriorates the predicting performance of subsequent learning algorithms. In this paper, motivated by the success of improving predicting performance by ensemble learning, we propose to enhance privacy-preserving logistic regression by stacking. We show that this can be done either by sample-based or feature-based partitioning. However, we prove that when privacy-budgets are the same, feature-based partitioning requires fewer samples than sample-based one, and thus likely has better empirical performance. As transfer learning is difficult to be integrated with a differential privacy guarantee, we further combine the proposed method with hypothesis transfer learning to address the problem of learning across different organizations. Finally, we not only demonstrate the effectiveness of our method on two benchmark data sets, i.e., MNIST and NEWS20, but also apply it into a real application of cross-organizational diabetes prediction from RUIJIN data set, where privacy is of significant concern.

Introduction

In the era of strict data privacy laws like GDPR, organizations face a significant challenge: how to share knowledge and build robust machine learning models without exposing sensitive raw data. Differential Privacy (DP) has emerged as the gold standard for protecting individual records by adding calibrated noise to learning algorithms. However, this introduces a "privacy-utility trade-off," where increased noise to ensure privacy leads to degraded model accuracy. Standard privacy-preserving logistic regression (PLR) often suffers when noise overwhelms the underlying signal.

The Privacy-preserving Stacking (PST) Framework

To address the performance degradation in DP models, this research proposes Privacy-preserving Stacking (PST). Stacking is an ensemble method that combines multiple "base" models using a high-level aggregator. The authors explore two primary partitioning strategies for this ensemble:

  1. Sample-based Partitioning (PST-S): The dataset is split by rows (samples). Each subset trains a private model, and their outputs are combined.
  2. Feature-based Partitioning (PST-F): The dataset is split by columns (features). Crucially, each low-level model utilizes the entire set of available samples, but only a subset of features.

The Advantage of Feature Partitioning

A key theoretical contribution of this work is the proof that feature-based partitioning (PST-F) is mathematically superior to sample-based partitioning under DP constraints. Because DP models are highly sensitive to sample size, PST-F's ability to provide every sub-model with the full sample size allows it to handle noise much more effectively.

Furthermore, PST-F enables Strategic Noise Allocation. By assigning importance weights to different feature groups (e.g., prioritizing blood glucose levels over age in a medical context), the algorithm can spend its "privacy budget" more wisely, preserving the signal in vital features while heavily protecting less relevant ones.

Empirical Performance and Results

The researchers evaluated PST against standard PLR and sample-based stacking on benchmark datasets (MNIST and NEWS20) and a massive real-world medical dataset (RUIJIN).

Performance comparison on MNIST: PST-F (weighted and unweighted) consistently outperforms sample-based stacking (PST-S) and standard private logistic regression (PLR) across various privacy budgets (epsilon).

As shown in the experimental results, PST-F consistently outperforms both standard PLR and PST-S. The high-level model in the stacking architecture acts as a robust aggregator, effectively smoothing out the noise introduced in the low-level private models.

Hyperparameter Sensitivity: The "K" Sweet Spot

The performance of PST-F is influenced by the number of partitions ($K$). The authors identified a "sweet spot" (typically around $K=5$) where the model achieves optimal balance. If $K$ is too low, the ensemble gain is insufficient; if $K$ is too high, the model loses critical correlations between features within the sub-models.

Impact of the number of partitions (K) on testing AUC. Performance peaks at moderate K values before declining as feature correlations are lost.

Cross-Organizational Transfer Learning

To facilitate learning across different organizations, the authors introduced PST-H, which integrates Hypothesis Transfer Learning. This allows a "target" organization to improve its private model by borrowing knowledge from a "source" organization's pre-trained models without any raw data exchange. In the RUIJIN hospital application—predicting diabetes across 16 different branches—PST-H significantly outperformed existing state-of-the-art private transfer learning methods.

Conclusion and Future Directions

The PST-F framework demonstrates that ensemble stacking, when applied via feature partitioning, can effectively mitigate the utility loss inherent in differential privacy. While currently focused on convex models like logistic regression, future work may extend these guarantees to non-convex models such as deep neural networks. Additionally, automating the feature importance weighting process—currently dependent on domain expertise—remains a promising area for further research.

Summary updated

Abstract

To meet the standard of differential privacy, noise is usually added into the original data, which inevitably deteriorates the predicting performance of subsequent learning algorithms. In this paper, motivated by the success of improving predicting performance by ensemble learning, we propose to enhance privacy-preserving logistic regression by stacking. We show that this can be done either by sample-based or feature-based partitioning. However, we prove that when privacy-budgets are the same, feature-based partitioning requires fewer samples than sample-based one, and thus likely has better empirical performance. As transfer learning is difficult to be integrated with a differential privacy guarantee, we further combine the proposed method with hypothesis transfer learning to address the problem of learning across different organizations. Finally, we not only demonstrate the effectiveness of our method on two benchmark data sets, i.e., MNIST and NEWS20, but also apply it into a real application of cross-organizational diabetes prediction from RUIJIN data set, where privacy is of significant concern.
BibTeX
@inproceedings{guo2019privacy,
  author = {Xiawei Guo and Quanming Yao and Wei-Wei Tu and Yuqiang Chen and Wenyuan Dai and Qiang Yang},
  title = {{Privacy-preserving Transfer Learning for Knowledge Sharing}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2019},
  doi = {10.24963/ijcai.2019/571},
}
Figure