Abstract
Training deep neural networks on sensitive data typically requires a trade-off between privacy and performance. Traditional approaches using Fully Homomorphic Encryption (FHE) suffer from significant computational overhead and accuracy degradation due to the necessity of polynomial approximations for non-linear activation functions. We present GELU-Net, a framework that partitions neural network training into a globally encrypted linear component handled by a server and a locally unencrypted non-linear component handled by the client. This hybrid approach maintains 100% of the original model's accuracy while achieving up to a 35x speedup over existing FHE-based systems like CryptoNets.
1. Introduction
The proliferation of deep learning in sensitive domains such as healthcare and finance has intensified the need for privacy-preserving training methods. While Fully Homomorphic Encryption (FHE) allows for computation on encrypted data, it introduces three primary challenges: extreme computational latency (Performance), the inability to compute non-linear activations without inaccurate polynomial approximations (Polynomials), and resulting training instabilities (Precision/Stability). GELU-Net addresses these issues by strategically splitting the computational workload between the data owner (client) and the computation provider (server).
2. Methodology
GELU-Net utilizes a Globally Encrypted, Locally Unencrypted strategy. Every neuron in the network is partitioned into two functional segments:
- Linear Component (Server-Side): The server performs weighted sums ($w \cdot a + b$) using Paillier encryption. Paillier is a partially homomorphic encryption scheme that is significantly more efficient than FHE but limited to addition and scalar multiplication.
- Non-linear Component (Client-Side): Since Paillier cannot compute non-linear activations, the server returns the encrypted intermediate results to the client. The client decrypts the data, applies the standard activation function (e.g., ReLU or Sigmoid) in plaintext, re-encrypts the result, and returns it to the server.
2.1 Privacy Protections
To prevent the server from inferring sensitive information from gradient updates or intermediate values, GELU-Net employs two key mechanisms:
- Random Masking: Clients inject random noise into gradients before transmission. Without such protections, the server could potentially reverse-engineer the input data from the gradient updates.
- Training Bounds ($d_{max}$): The server limits the number of continuous training iterations a single client can perform to prevent the client from solving systems of linear equations to extract server-side model weights.
3. Experimental Evaluation
The framework was evaluated on four benchmarks: Iris, Diabetes, kr-vs-kp, and MNIST. GELU-Net was compared against state-of-the-art privacy-preserving models, including CryptoNets (FHE-based) and BGN-Net.
4. Results and Discussion
GELU-Net demonstrates superior performance across all metrics:
- Accuracy: Unlike models that rely on polynomial approximations, GELU-Net achieves the exact same accuracy as non-private baselines. On the MNIST dataset, GELU-Net maintained ~97% accuracy, whereas competing private models dropped to 90-91%.
- Efficiency: The system is 14x to 35x faster than CryptoNets. The overhead of network communication for client-side activations is significantly lower than the computational cost of FHE-based non-linear approximations.
- Stability: GELU-Net avoids the "loss surges" common in polynomial-based models, ensuring stable convergence during training.
As shown in the figures above, the random masking effectively prevents high-fidelity reconstruction of the input data (e.g., MNIST digits), even as the number of hidden neurons varies. This demonstrates that the communication-based approach to non-linearities does not compromise the underlying data privacy.
5. Conclusion
GELU-Net provides a practical path forward for privacy-preserving deep learning by leveraging the efficiency of partially homomorphic encryption and the flexibility of local plaintext activations. While the current implementation assumes a semi-honest security model and faces potential communication bottlenecks in extremely deep or wide networks, it represents a significant leap in balancing privacy, accuracy, and performance. Future work will explore robustness against malicious actors and scaling to modern transformer-based architectures.
Summary updated