TL;DR
This paper introduces NeuJeans, a novel system for private neural network inference using fully homomorphic encryption (FHE). NeuJeans significantly speeds up FHE-based convolutional neural network (CNN) inference by introducing a new encoding method (CinS) and optimizing the interaction between convolution operations and FHE bootstrapping, achieving near real-time inference for complex models.
What Problem Does This Paper Address?
The increasing adoption of machine learning (ML) in sensitive domains like healthcare and finance raises privacy concerns. While private inference (PI) allows users to offload ML tasks to cloud servers without revealing their data or the proprietary model, existing FHE-based PI solutions for deep neural networks, particularly CNNs, are computationally prohibitive. The core challenges are the enormous cost of FHE operations, especially for complex computations like convolutions, and the overhead of bootstrapping (a process to reset FHE operation limits). Current methods struggle with long inference times, often taking minutes or hours for a single inference, hindering real-world adoption.
Methods
NeuJeans tackles these challenges with several key innovations:
- Coefficients-in-Slot (CinS) Encoding: This novel encoding method allows multiple convolutions to be performed within a single HE multiplication without expensive data rearrangements (slot permutations). It leverages properties of the Discrete Fourier Transform (DFT) to enable efficient local convolutions.
- Optimized Convolution (conv2d) Algorithms: By using CinS encoding, NeuJeans devises specialized algorithms for various 2D convolution operations. These algorithms reduce the number of multiplications and rotations required, significantly improving efficiency.
- Fusing Convolution with Bootstrapping: The authors observe that the conversion steps required during bootstrapping (specifically, converting between different encoding types) share similarities with operations needed for convolution. They exploit this to fuse these operations, effectively making the conversion cost free during the inference process.
- FHE-Friendly Execution Flows: NeuJeans carefully designs the order of operations within CNN layers to minimize expensive bootstrapping and rotation operations, especially for downsampling and depthwise convolutions. This includes a novel decomposition method for downsampling convolutions to avoid processing sparsely packed ciphertexts.
Datasets
The paper evaluates NeuJeans on the ImageNet dataset. They train and test CNN models like ResNet18, ResNet50, and MobileNetV2 on this dataset.
Key Insights and Ideas
- The Bottleneck is Convolution and Bootstrapping: The paper clearly identifies that the computational cost of convolutions and the overhead of bootstrapping are the primary bottlenecks in FHE-based CNN inference.
- Encoding is Key to Efficiency: The choice of data encoding within FHE ciphertexts dramatically impacts the efficiency of operations. NeuJeans's CinS encoding is a significant improvement over existing methods by enabling more efficient convolution.
- Synergy Between Operations: The most impactful insight is the discovery of how to fuse convolution operations with the bootstrapping process. This "free" conversion significantly reduces overall computation.
- Optimizing for Specific Operations: By tailoring algorithms and execution flows for different types of convolutions (default, downsampling, depthwise), NeuJeans achieves substantial speedups.
Key Results
- Significant Speedups: NeuJeans achieves speedups of up to 5.68x for conv2d-activation sequences compared to state-of-the-art FHE-based PI work.
- Near Real-Time Inference: For ImageNet-scale CNNs like ResNet18, NeuJeans can perform private inference in as little as 5.35 seconds on a GPU-accelerated server. This is a dramatic improvement over previous methods that took minutes to hours.
- Accuracy Preservation: For models trained with specialized methods (like AESPA), NeuJeans maintains comparable accuracy to the original models. For models using polynomial approximations for activation, a slight drop in accuracy (2.3-2.4%) was observed, which is attributed to the approximation itself rather than the FHE process.
- Reduced Client Overhead: Client-side tasks (encryption/decryption) are minimal, adding only dozens of milliseconds to the end-to-end inference time, with a total data communication of around 7MB for a single inference.
Limitations and Future Work
- Activation Function Overhead: While NeuJeans significantly optimizes convolutions, activation functions (especially non-polynomial ones requiring high-degree polynomial approximations) still contribute a notable portion to the overall inference time. The paper notes that models trained with specialized methods (like AESPA) that use low-degree polynomial activations experience larger speedups because activation becomes a smaller fraction of the total computation.
- Accuracy Drop with Polynomial Approximations: As mentioned, using polynomial approximations for activation functions can lead to a slight accuracy degradation.
- Future Work: The authors suggest that further optimizations could be achieved by exploring more efficient activation function approximations and potentially hardware acceleration for FHE operations. The paper also hints at the possibility of extending the fusion technique to other linear layers within CNNs.
Summary updated