TL;DR
This paper introduces HyPHEN, a novel framework for performing Convolutional Neural Network (CNN) inference using Fully Homomorphic Encryption (FHE). HyPHEN significantly reduces the computational and memory overhead of existing FHE-based CNNs by introducing new convolution algorithms and data packing techniques, achieving practical inference times for CIFAR-10 and demonstrating the first FHE-based inference on ImageNet.
What Problem Does This Paper Address?
The core challenge addressed is making private inference (PI) using Fully Homomorphic Encryption (FHE) practical for Convolutional Neural Networks (CNNs). While FHE allows computations on encrypted data, enabling secure offloading of inference to a server without revealing sensitive user data or the model itself, existing FHE-based CNN (HCNN) approaches suffer from extremely high computational and memory overhead. This makes them impractical for real-world applications, especially for larger datasets like ImageNet. The authors highlight that the manipulation of data organization in FHE is particularly costly, leading to frequent and expensive operations like ciphertext rotation and bootstrapping, which are major bottlenecks.
Methods
HyPHEN tackles the HCNN challenges through a combination of novel techniques:
- Novel Convolution Algorithms (RAConv and CAConv): These algorithms are designed to streamline data arrangement between consecutive convolutions, reducing the need for costly data reordering operations.
- Data Packing Methods (2D Gap Packing and PRCR Scheme):
- 2D Gap Packing: This method collaboratively uses duplication and channel multiplexing to alleviate the high cost of repacking data within ciphertexts, particularly for handling gaps introduced by strided convolutions.
- PRCR (Plaintext size Reduction through the image to Ciphertext Rearrangement): This technique addresses the significant memory footprint of weight plaintexts, especially for high-resolution images. It involves splitting images into segments and rearranging weight plaintexts to reduce their size and enable reuse, thereby mitigating memory constraints.
- Optimization Techniques: HyPHEN integrates these methods to minimize expensive homomorphic operations like ciphertext rotation and bootstrapping. This includes reordering operations within RAConv to reduce rotation costs and carefully placing bootstrapping to occur when the number of ciphertexts is minimal.
Datasets
- CIFAR-10: Used for training and evaluating ResNet-20/32/44 models.
- ImageNet: Used for training and evaluating the ResNet-18 model.
Key Insights and Ideas
- Data organization is paramount in FHE: The authors emphasize that the way data is arranged within ciphertexts has a disproportionately large impact on FHE performance, more so than the cost of individual operations.
- Balancing slot usage and rotation cost: Dense packing of data into ciphertexts maximizes slot utilization but can increase rotation costs, while sparse packing reduces rotations but wastes slots. HyPHEN aims to find an optimal balance.
- PRCR for memory-constrained environments: The significant memory overhead of weight plaintexts for large datasets is a critical bottleneck. PRCR offers a practical solution by reducing this footprint, making FHE-CNN feasible on hardware with limited memory.
- AESPA for efficient activation: Using low-degree polynomial approximations for activation functions (like AESPA) significantly reduces the computational cost and level consumption, making bootstrapping less frequent.
Key Results
- CIFAR-10 Inference: HyPHEN achieves a practical inference time of 1.4 seconds for ResNet-20 on CIFAR-10.
- ImageNet Inference: HyPHEN demonstrates the first end-to-end FHE-based CNN inference on the ImageNet dataset with ResNet-18, achieving an execution time of 14.69 seconds.
- Memory Reduction: The PRCR scheme provides substantial memory reduction, achieving 6.15-6.81× memory reduction for ResNet-18 on ImageNet compared to a baseline.
- Speedup: HyPHEN offers significant speedups, with 1.83×/2.15× lower latency compared to prior state-of-the-art for ResNet-20/18 on CPU. GPU acceleration further improves performance.
- Accuracy: HyPHEN maintains near-zero accuracy degradation (< 0.01%) for CIFAR-10 models and shows comparable accuracy to the backbone for ResNet-18 on ImageNet.
Limitations and Future Work
- Inference Time: While significantly reduced, a few seconds for inference is still considered challenging for some real-world deployment scenarios.
- Memory Capacity for Evolving Models: As image classification models become more complex and require larger memory capacities, further advancements in memory footprint reduction algorithms will be necessary.
- Future Work: The authors suggest that further research into advanced memory footprint reduction algorithms could facilitate broader adoption of Private Protected Machine Learning (PPML). They also note that specialized FHE hardware accelerators could offer orders of magnitude performance improvements.
Summary updated