Differential Private Stack Generalization with an Application to Diabetes Prediction

Quanming Yao, Xiawei Guo, James T. Kwok, Wei-Wei Tu, Yuqiang Chen, Wenyuan Dai, Qiang Yang

Abstract

To meet the standard of differential privacy, noise is usually added into the original data, which inevitably deteriorates the predicting performance of subsequent learning algorithms. In this paper, motivated by the success of improving predicting performance by ensemble learning, we propose to enhance privacy-preserving logistic regression by stacking. We show that this can be done either by sample-based or feature-based partitioning. However, we prove that when privacy-budgets are the same, feature-based partitioning requires fewer samples than sample-based one, and thus likely has better empirical performance. As transfer learning is difficult to be integrated with a differential privacy guarantee, we further combine the proposed method with hypothesis transfer learning to address the problem of learning across different organizations. Finally, we not only demonstrate the effectiveness of our method on two benchmark data sets, i.e., MNIST and NEWS20, but also apply it into a real application of cross-organizational diabetes prediction from RUIJIN data set, where privacy is of significant concern.

Introduction

Differential Privacy (DP) has emerged as a rigorous framework for protecting individual privacy in data analysis. However, achieving DP often incurs a "utility tax"—the injection of noise required to satisfy privacy guarantees typically degrades a model's predictive accuracy. This challenge is particularly acute in sensitive domains like healthcare, where cross-organizational collaboration (e.g., between multiple hospitals) is hindered by strict data protection regulations like GDPR.

This paper addresses the utility-privacy trade-off by introducing Privacy-preserving Stacking (PST). By leveraging ensemble learning—specifically stacking—the authors aim to reclaim the accuracy lost to DP noise without compromising privacy.

Methodology: Privacy-preserving Stacking

Stacking involves training multiple "low-level" models on subsets of data and then employing a "high-level" aggregator to combine their predictions. The authors explore two primary partitioning strategies:

  1. Sample-based Partitioning (PST-S): The dataset is split by rows, with different models trained on different groups of individuals.
  2. Feature-based Partitioning (PST-F): The dataset is split by columns, where each model focuses on a specific subset of attributes (e.g., lab results vs. demographics).

A key theoretical contribution of this work is the proof that feature-based partitioning is superior under DP constraints. In PST-F, each low-level model observes all available samples, albeit with fewer features. This leads to lower sample complexity compared to PST-S to achieve a target accuracy level.

To handle cross-organizational scenarios, the authors further propose PST-H, which integrates Hypothesis Transfer Learning (HTL). This allows a target organization to utilize a pre-trained model from a source organization as a regularizer, effectively "transferring" knowledge privately.

Experimental Evaluation

The proposed methods were evaluated on benchmark datasets (MNIST and NEWS20) and a massive real-world dataset (RUIJIN) containing over 100,000 diabetes records from 16 hospital branches.

Testing AUC vs. privacy budget epsilon for MNIST and NEWS20 datasets.

As shown in the results above, PST-F consistently outperforms both standard private logistic regression (PLR) and sample-based stacking (PST-S) across various privacy budgets. Furthermore, incorporating domain expertise to weight feature importance (PST-F(W)) yields significant performance gains over uniform weighting (PST-F(U)). In the RUIJIN diabetes application, PST-H demonstrated superior performance in cross-organizational prediction, successfully improving local branch models by leveraging knowledge from the main hospital.

Discussion and Limitations

While PST-F offers clear advantages, its performance is sensitive to the number of partitions ($K$).

Testing AUC vs. number of partitions K on the MNIST dataset.

As $K$ increases, the model may lose critical correlations between features placed in different groups, potentially leading to a performance plateau or decline. Additionally, while this work focuses on convex models like logistic regression for their stability, extending the stacking framework to non-convex architectures like Deep Neural Networks remains an open area for future research.

Conclusion

By reimagining stacking through the lens of feature partitioning and smart noise allocation, this work provides a robust framework for high-utility, privacy-preserving machine learning. The success on the RUIJIN dataset underscores its practical potential for real-world clinical applications where data privacy is paramount.

Summary updated

Abstract

To meet the standard of differential privacy, noise is usually added into the original data, which inevitably deteriorates the predicting performance of subsequent learning algorithms. In this paper, motivated by the success of improving predicting performance by ensemble learning, we propose to enhance privacy-preserving logistic regression by stacking. We show that this can be done either by sample-based or feature-based partitioning. However, we prove that when privacy-budgets are the same, feature-based partitioning requires fewer samples than sample-based one, and thus likely has better empirical performance. As transfer learning is difficult to be integrated with a differential privacy guarantee, we further combine the proposed method with hypothesis transfer learning to address the problem of learning across different organizations. Finally, we not only demonstrate the effectiveness of our method on two benchmark data sets, i.e., MNIST and NEWS20, but also apply it into a real application of cross-organizational diabetes prediction from RUIJIN data set, where privacy is of significant concern.
BibTeX
@inproceedings{yao2019differential,
  author = {Quanming Yao and Xiawei Guo and James T. Kwok and Wei-Wei Tu and Yuqiang Chen and Wenyuan Dai and Qiang Yang},
  title = {{Differential Private Stack Generalization with an Application to Diabetes Prediction}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2019},
  doi = {10.24963/ijcai.2019/571},
}
Figure