GELU-Net: A Globally Encrypted, Locally Unencrypted Deep Neural Network for Privacy-Preserved Learning

Qiao Zhang, Cong Wang, Hongyi Wu, Chunsheng Xin, Tran V. PhuongIJCAI 2018 · DOI

Abstract

Training deep neural networks on sensitive data typically requires a trade-off between privacy and performance. Traditional approaches using Fully Homomorphic Encryption (FHE) suffer from significant computational overhead and accuracy degradation due to the necessity of polynomial approximations for non-linear activation functions. We present GELU-Net, a framework that partitions neural network training into a globally encrypted linear component handled by a server and a locally unencrypted non-linear component handled by the client. This hybrid approach maintains 100% of the original model's accuracy while achieving up to a 35x speedup over existing FHE-based systems like CryptoNets.

1. Introduction

The proliferation of deep learning in sensitive domains such as healthcare and finance has intensified the need for privacy-preserving training methods. While Fully Homomorphic Encryption (FHE) allows for computation on encrypted data, it introduces three primary challenges: extreme computational latency (Performance), the inability to compute non-linear activations without inaccurate polynomial approximations (Polynomials), and resulting training instabilities (Precision/Stability). GELU-Net addresses these issues by strategically splitting the computational workload between the data owner (client) and the computation provider (server).

2. Methodology

GELU-Net utilizes a Globally Encrypted, Locally Unencrypted strategy. Every neuron in the network is partitioned into two functional segments:

  • Linear Component (Server-Side): The server performs weighted sums ($w \cdot a + b$) using Paillier encryption. Paillier is a partially homomorphic encryption scheme that is significantly more efficient than FHE but limited to addition and scalar multiplication.
  • Non-linear Component (Client-Side): Since Paillier cannot compute non-linear activations, the server returns the encrypted intermediate results to the client. The client decrypts the data, applies the standard activation function (e.g., ReLU or Sigmoid) in plaintext, re-encrypts the result, and returns it to the server.

2.1 Privacy Protections

To prevent the server from inferring sensitive information from gradient updates or intermediate values, GELU-Net employs two key mechanisms:

  1. Random Masking: Clients inject random noise into gradients before transmission. Without such protections, the server could potentially reverse-engineer the input data from the gradient updates.
  2. Training Bounds ($d_{max}$): The server limits the number of continuous training iterations a single client can perform to prevent the client from solving systems of linear equations to extract server-side model weights.
Reconstruction attempt without privacy protections.

3. Experimental Evaluation

The framework was evaluated on four benchmarks: Iris, Diabetes, kr-vs-kp, and MNIST. GELU-Net was compared against state-of-the-art privacy-preserving models, including CryptoNets (FHE-based) and BGN-Net.

4. Results and Discussion

GELU-Net demonstrates superior performance across all metrics:

  • Accuracy: Unlike models that rely on polynomial approximations, GELU-Net achieves the exact same accuracy as non-private baselines. On the MNIST dataset, GELU-Net maintained ~97% accuracy, whereas competing private models dropped to 90-91%.
  • Efficiency: The system is 14x to 35x faster than CryptoNets. The overhead of network communication for client-side activations is significantly lower than the computational cost of FHE-based non-linear approximations.
  • Stability: GELU-Net avoids the "loss surges" common in polynomial-based models, ensuring stable convergence during training.
Privacy-preserved reconstruction attempts across different model configurations.

As shown in the figures above, the random masking effectively prevents high-fidelity reconstruction of the input data (e.g., MNIST digits), even as the number of hidden neurons varies. This demonstrates that the communication-based approach to non-linearities does not compromise the underlying data privacy.

5. Conclusion

GELU-Net provides a practical path forward for privacy-preserving deep learning by leveraging the efficiency of partially homomorphic encryption and the flexibility of local plaintext activations. While the current implementation assumes a semi-honest security model and faces potential communication bottlenecks in extremely deep or wide networks, it represents a significant leap in balancing privacy, accuracy, and performance. Future work will explore robustness against malicious actors and scaling to modern transformer-based architectures.

Summary updated

BibTeX
@inproceedings{zhang2018gelu,
  author = {Qiao Zhang and Cong Wang and Hongyi Wu and Chunsheng Xin and Tran V. Phuong},
  title = {{GELU-Net: A Globally Encrypted, Locally Unencrypted Deep Neural Network for Privacy-Preserved Learning}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2018},
  doi = {10.24963/ijcai.2018/547},
}
Figure