TL;DR
This paper introduces SMART-PAF, a framework that significantly improves the accuracy and speed of machine learning inference on encrypted data using homomorphic encryption. By developing novel training techniques to accurately approximate non-polynomial operations with low-degree polynomials, SMART-PAF achieves substantial gains in both accuracy and latency compared to prior methods.
What Problem Does This Paper Address?
Machine learning (ML) is increasingly used in sensitive domains like healthcare and facial recognition, making data privacy paramount. Fully Homomorphic Encryption (FHE) offers a solution by allowing computations on encrypted data, thus preserving privacy. However, FHE introduces a massive performance bottleneck, slowing down inference by up to five orders of magnitude. A primary culprit is the need to approximate non-polynomial operations (like ReLU and MaxPooling) with high-degree polynomials, which are computationally expensive in FHE. Existing methods struggle to balance accuracy and speed, often resulting in significant accuracy degradation or requiring prohibitively complex polynomial approximations.
Methods
The authors propose a framework called SMART-PAF, which combines four novel techniques to address the challenges of approximating non-polynomial operators in FHE:
- Coefficient Tuning (CT): This pre-training technique adjusts polynomial approximation function (PAF) coefficients based on the specific input distributions of different layers. This leads to better initial approximations and faster convergence.
- Progressive Approximation (PA): Instead of replacing all non-polynomial operators at once, PA replaces them one by one, followed by fine-tuning. This sequential approach simplifies the optimization problem and guarantees convergence under Stochastic Gradient Descent (SGD).
- Alternate Training (AT): This technique decouples the training of PAF coefficients from the training of other linear layers (like convolutions). By using different hyperparameters and alternating between training these two components, AT prevents interference and improves accuracy.
- Dynamic Scaling (DS) and Static Scaling (SS): DS normalizes input values to a fixed range ([-1, 1]) during training to improve PAF approximation. For FHE deployment, SS fixes this scale to the maximum input value observed during training, preventing issues with value-dependent operations in FHE.
These techniques are orchestrated by a scheduler within the SMART-PAF framework, which systematically applies them to achieve optimal results.
Datasets
The paper evaluates SMART-PAF on two standard datasets:
- ImageNet-1k: A large-scale dataset commonly used for image classification.
- CiFar-10: A smaller dataset also used for image classification.
The models used for evaluation are VGG-19 and ResNet-18, which are standard architectures for these datasets.
Key Insights and Ideas
- Non-polynomial operators are the primary bottleneck: The paper clearly identifies non-polynomial operations as the main cause of latency in FHE-based ML inference.
- Low-degree PAFs are feasible with proper training: Contrary to the belief that high-degree PAFs are necessary for accuracy, SMART-PAF demonstrates that low-degree PAFs can achieve competitive or even superior results when combined with advanced training techniques.
- Input distribution matters: The effectiveness of PAFs is highly dependent on the input data distribution. CT leverages this by tailoring approximations to specific layer inputs.
- Sequential training is crucial for convergence: The progressive replacement and training approach (PA) is key to overcoming the convergence issues that plague direct replacement methods.
- Decoupling training components is beneficial: Separating the training of PAFs and linear layers (AT) addresses the differing needs of these components and leads to better accuracy.
- Scaling is critical for both training and deployment: DS and SS address the challenge of input range management, which is vital for both accurate training and FHE compatibility.
Key Results
- Significant accuracy improvement: SMART-PAF achieves accuracy improvements of 1.42× to 13.64× for ResNet-18 on ImageNet-1k compared to prior works.
- Substantial speedup: The framework offers speedups of 6.79× to 14.9× for ResNet-18 on ImageNet-1k.
- Low-degree PAFs outperform high-degree ones: SMART-PAF enables a 14-degree PAF to achieve comparable accuracy to a 27-degree PAF from prior work, but with a 7.81× speedup.
- Improved accuracy on VGG-19: For VGG-19 on CiFar-10, SMART-PAF provides accuracy enhancements ranging from 1.01× to 1.75×.
- Mitigation of accuracy degradation: The proposed techniques effectively reduce the accuracy drop typically associated with PAF approximation.
Limitations and Future Work
- Dataset dependency: The paper notes that the effectiveness of PAF selection can be task-dependent, with more complex datasets like ImageNet-1k showing a greater impact from PAF choices compared to simpler datasets like CiFar-10.
- Future work: The authors suggest that their work "potentially opens a new paradigm of systematically augmenting PAF-approximated ML models for accurate and fast private inference." Further exploration into optimizing the framework for different model architectures and FHE schemes could be a natural next step. The paper also mentions that AT's effectiveness can vary across different layers, suggesting potential for more nuanced application of this technique.
Summary updated