Abstract
To meet the standard of differential privacy, noise is usually added into the original data, which inevitably deteriorates the predicting performance of subsequent learning algorithms. In this paper, motivated by the success of improving predicting performance by ensemble learning, we propose to enhance privacy-preserving logistic regression by stacking. We show that this can be done either by sample-based or feature-based partitioning. However, we prove that when privacy-budgets are the same, feature-based partitioning requires fewer samples than sample-based one, and thus likely has better empirical performance. As transfer learning is difficult to be integrated with a differential privacy guarantee, we further combine the proposed method with hypothesis transfer learning to address the problem of learning across different organizations. Finally, we not only demonstrate the effectiveness of our method on two benchmark data sets, i.e., MNIST and NEWS20, but also apply it into a real application of cross-organizational diabetes prediction from RUIJIN data set, where privacy is of significant concern.
Introduction
Differential Privacy (DP) has emerged as a rigorous framework for protecting individual privacy in data analysis. However, achieving DP often incurs a "utility tax"—the injection of noise required to satisfy privacy guarantees typically degrades a model's predictive accuracy. This challenge is particularly acute in sensitive domains like healthcare, where cross-organizational collaboration (e.g., between multiple hospitals) is hindered by strict data protection regulations like GDPR.
This paper addresses the utility-privacy trade-off by introducing Privacy-preserving Stacking (PST). By leveraging ensemble learning—specifically stacking—the authors aim to reclaim the accuracy lost to DP noise without compromising privacy.
Methodology: Privacy-preserving Stacking
Stacking involves training multiple "low-level" models on subsets of data and then employing a "high-level" aggregator to combine their predictions. The authors explore two primary partitioning strategies:
- Sample-based Partitioning (PST-S): The dataset is split by rows, with different models trained on different groups of individuals.
- Feature-based Partitioning (PST-F): The dataset is split by columns, where each model focuses on a specific subset of attributes (e.g., lab results vs. demographics).
A key theoretical contribution of this work is the proof that feature-based partitioning is superior under DP constraints. In PST-F, each low-level model observes all available samples, albeit with fewer features. This leads to lower sample complexity compared to PST-S to achieve a target accuracy level.
To handle cross-organizational scenarios, the authors further propose PST-H, which integrates Hypothesis Transfer Learning (HTL). This allows a target organization to utilize a pre-trained model from a source organization as a regularizer, effectively "transferring" knowledge privately.
Experimental Evaluation
The proposed methods were evaluated on benchmark datasets (MNIST and NEWS20) and a massive real-world dataset (RUIJIN) containing over 100,000 diabetes records from 16 hospital branches.
As shown in the results above, PST-F consistently outperforms both standard private logistic regression (PLR) and sample-based stacking (PST-S) across various privacy budgets. Furthermore, incorporating domain expertise to weight feature importance (PST-F(W)) yields significant performance gains over uniform weighting (PST-F(U)). In the RUIJIN diabetes application, PST-H demonstrated superior performance in cross-organizational prediction, successfully improving local branch models by leveraging knowledge from the main hospital.
Discussion and Limitations
While PST-F offers clear advantages, its performance is sensitive to the number of partitions ($K$).
As $K$ increases, the model may lose critical correlations between features placed in different groups, potentially leading to a performance plateau or decline. Additionally, while this work focuses on convex models like logistic regression for their stability, extending the stacking framework to non-convex architectures like Deep Neural Networks remains an open area for future research.
Conclusion
By reimagining stacking through the lens of feature partitioning and smart noise allocation, this work provides a robust framework for high-utility, privacy-preserving machine learning. The success on the RUIJIN dataset underscores its practical potential for real-world clinical applications where data privacy is paramount.
Summary updated