Industrial Scale Privacy Preserving Deep Neural Network

Longfei Zheng, Chaochao Chen, Yingting Liu, Bingzhe Wu, Xibin Wu, Li Wang, Lei Wang, Jun Zhou, Shuang YangarXiv 2020 · OpenReview

Abstract

Deep Neural Networks (DNNs) are powerful tools for applications like fraud detection, but "data isolation"—the inability of parties to share sensitive data—remains a significant barrier. While cryptographic techniques offer privacy, they often fail to scale to industrial datasets. This paper introduces $P^2N^2$, a privacy-preserving neural network learning paradigm that balances security and scalability. By splitting the computation graph and employing a novel "defender" mechanism, $P^2N^2$ enables collaborative training on millions of records with minimal accuracy loss and high efficiency.

Introduction: The Data Isolation Challenge

In sectors like finance and insurance, data is often siloed. While combining datasets from multiple institutions could significantly improve model performance, strict privacy regulations like GDPR prevent direct sharing. Traditional cryptographic solutions, such as Homomorphic Encryption (HE) or Secure Multi-Party Computation (MPC), provide strong security but introduce massive computational and communication overheads, making them impractical for deep learning on industrial-scale datasets.

The $P^2N^2$ Framework

The proposed Privacy Preserving Neural Network ($P^2N^2$) architecture addresses the scalability-privacy trade-off through a hybrid approach.

Computation Splitting and Secret Sharing

The core idea is to divide the DNN's computation graph into private and public components. Data holders locally process the input layers and the final label/prediction layers. The computationally intensive middle hidden layers are offloaded to a neutral third-party server. To protect raw features during the initial combination, the first hidden layer is computed via secret sharing, ensuring that no party reveals their raw data to others.

The $P^2N^2$ architecture showing split computation between data holders and a neutral server, incorporating defender mechanisms.

The Defender Mechanism

A critical risk in split learning is "inversion attacks," where a curious server might attempt to reconstruct raw inputs from the hidden layer activations it receives. $P^2N^2$ introduces an adversarial defender mechanism. During training, the model is optimized not only for the primary task (e.g., fraud detection) but also to minimize the ability of an internal "attacker" to reconstruct the original input. This adversarial loss forces the network to learn representations that are task-relevant but privacy-preserving.

Experimental Evaluation

The framework was tested on real-world datasets, including Kaggle's Fraud Detection and Financial Distress datasets, as well as MNIST for visual verification.

Key Results

  • Accuracy: On the Fraud Detection task, $P^2N^2$ achieved an AUC of 0.9231, nearly matching the 0.9270 achieved by a standard non-private model.
  • Scalability: While the secret sharing step makes $P^2N^2$ slower than plaintext training, it remains orders of magnitude faster than pure cryptographic methods, proving its viability for industrial use.
  • Training Stability: The model demonstrates smooth convergence, as shown in the training and test loss curves.
Training and test loss curves demonstrating the convergence and stability of the $P^2N^2$ model over 100 iterations.

Conclusion

$P^2N^2$ represents a practical step forward for privacy-preserving machine learning in industry. By strategically applying cryptographic techniques and leveraging adversarial training for privacy, it overcomes the scalability hurdles of traditional methods. Future work will focus on hardening the system against malicious (rather than just semi-honest) actors and deploying the framework in live production environments at Ant Financial.

Summary updated

Abstract

Deep Neural Network (DNN) has been showing great potential in kinds of real-world applications such as fraud detection and distress prediction. Meanwhile, data isolation has become a serious problem currently, i.e., different parties cannot share data with each other. To solve this issue, most research leverages cryptographic techniques to train secure DNN models for multi-parties without compromising their private data. Although such methods have strong security guarantee, they are difficult to scale to deep networks and large datasets due to its high communication and computation complexities. To solve the scalability of the existing secure Deep Neural Network (DNN) in data isolation scenarios, in this paper, we propose an industrial scale privacy preserving neural network learning paradigm, which is secure against semi-honest adversaries. Our main idea is to split the computation graph of DNN into two parts, i.e., the computations related to private data are performed by each party using cryptographic techniques, and the rest computations are done by a neutral server with high computation ability. We also present a defender mechanism for further privacy protection. We conduct experiments on real-world fraud detection dataset and financial distress prediction dataset, the encouraging results demonstrate the practicalness of our proposal.
BibTeX
@misc{zheng2020industrial,
  author = {Longfei Zheng and Chaochao Chen and Yingting Liu and Bingzhe Wu and Xibin Wu and Li Wang and Lei Wang and Jun Zhou and Shuang Yang},
  title = {{Industrial Scale Privacy Preserving Deep Neural Network}},
  year = {2020},
  eprint = {2003.05198},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2003.05198},
}
Figure