TL;DR
This paper presents a novel approach to running deep convolutional neural networks (CNNs) on encrypted data using fully homomorphic encryption (FHE). By introducing multiplexed parallel convolutions and an "imaginary-removing bootstrapping" technique, they achieve significant speedups and enable the first-ever implementation of deep ResNet models (up to ResNet-110) on FHE with high accuracy, paving the way for practical privacy-preserving machine learning.
What Problem Does This Paper Address?
The increasing concern over data privacy, especially for sensitive information like medical records, has driven research into Privacy-Preserving Machine Learning (PPML). A key challenge is performing computations on encrypted data without decrypting it. While Fully Homomorphic Encryption (FHE) offers the theoretical capability to do this, its practical application to complex models like deep Convolutional Neural Networks (CNNs) has been severely limited by extremely high computational overhead, particularly the "bootstrapping" operation required to refresh ciphertexts. Previous attempts often relied on non-standard, shallow CNNs or required significant data communication, making them impractical for real-world use with standard, pre-trained deep models. This paper aims to bridge this gap by enabling efficient FHE-based inference for standard, deep CNNs like ResNets.
Methods
The authors introduce several key innovations to tackle the computational burden of FHE for deep CNNs:
- Multiplexed Parallel Convolution: Instead of processing data channel by channel, this method packs data from multiple channels into a single ciphertext. This significantly reduces the number of operations and, crucially, the number of expensive bootstrapping operations required. They also propose a parallel version of this convolution that further optimizes performance.
- Imaginary-Removing Bootstrapping: A critical issue identified is the divergence of approximate ReLU operations in deep networks when noise accumulates in the imaginary part of complex numbers used in FHE. This novel bootstrapping technique specifically targets and removes this imaginary noise before the ReLU approximation, preventing catastrophic failures and maintaining accuracy.
- Level Consumption Optimization: The authors integrate computations like convolution and batch normalization to reduce the "level consumption" (a measure of computational depth) of the FHE circuit, further minimizing the need for bootstrapping.
- Parameter Optimization: They fine-tune various parameters of the RNS-CKKS scheme and the network architecture to achieve a balance between security (128-bit) and performance.
Datasets
The paper uses the CIFAR-10 and CIFAR-100 datasets for evaluation. These are standard image classification datasets, commonly used for benchmarking CNNs. The authors utilize pre-trained parameters for ResNet models on these datasets.
Key Insights and Ideas
- Bootstrapping is the Bottleneck: The paper clearly identifies bootstrapping as the primary performance bottleneck in FHE for deep networks, motivating their focus on reducing its frequency and cost.
- Imaginary Noise is a Catastrophic Failure Mode: The discovery that accumulated imaginary noise in FHE can cause approximate ReLU functions to diverge is a significant finding. The proposed "imaginary-removing bootstrapping" directly addresses this, enabling the use of deep networks.
- Data Packing is Key to Efficiency: The success of their approach hinges on efficiently packing multiple data channels into single ciphertexts, drastically reducing the overhead of FHE operations.
- Deep ResNets are Feasible with FHE: The paper demonstrates that it's not only possible but practical to run deep, standard CNN architectures like ResNet-110 on encrypted data with high accuracy, a feat not achieved before.
Key Results
- Latency Reduction: The implementation of ResNet-20 achieves a 4.67x lower inference latency compared to previous state-of-the-art work, even when using only a single CPU thread (compared to 64 threads in the prior work).
- Amortized Runtime Improvement: The amortized runtime (per image) for ResNet-20 is reduced by an impressive 134x.
- Deep Network Implementation: For the first time, ResNet-32, ResNet-44, ResNet-56, and ResNet-110 are successfully implemented on the RNS-CKKS scheme with high accuracies, comparable to their non-encrypted counterparts.
- Security: The proposed architecture achieves standard 128-bit security.
Limitations and Future Work
- Single CPU Thread Performance: While the single-thread performance is a significant improvement, the authors note that further speedups can be expected with GPU or hardware accelerator implementations, suggesting future work in this direction.
- Dataset Size: The evaluation is primarily on CIFAR-10/100. Extending to larger datasets like ImageNet would be a valuable next step.
- Interactive vs. Non-Interactive PPML: The paper focuses on non-interactive PPML (using only FHE). Future work could explore hybrid approaches combining FHE with secure multi-party computation (MPC) for specific use cases.
Summary updated