TL;DR
This paper introduces AESPA, a novel low-degree polynomial activation function that significantly speeds up private inference by replacing computationally expensive ReLU activations with a more efficient polynomial approximation. AESPA achieves comparable accuracy to standard ReLU-based models while reducing online latency and communication costs by up to 42.1x and 28.3x, respectively, in hybrid private inference protocols.
What Problem Does This Paper Address?
Private inference (PI) allows machine learning models to be used without revealing sensitive user data or the model itself. Hybrid PI protocols, which combine multi-party computation (MPC) and homomorphic encryption (HE), are a promising approach. However, a major bottleneck in these protocols is the processing of non-linear activation functions, particularly ReLU. While ReLU offers high accuracy, it requires slow and communication-heavy "garbled circuit" (GC) MPC primitives. Polynomial approximations of ReLU are faster, using "Beaver's multiplication triples" (BT) MPC primitives, but they typically lead to a significant drop in model accuracy. This paper tackles the challenge of achieving both high accuracy and fast inference for non-linear activations in private inference settings.
Methods
The core innovation is the AESPA (Accuracy Preserving Low-degree Polynomial Activation) function. AESPA is built upon two key ideas:
- Hermite Polynomial Expansion: Instead of traditional Taylor series or minimax approximations, AESPA uses Hermite polynomials. These are orthogonal polynomials that, due to their Gaussian weight function and ordered nature, are well-suited for representing functions like ReLU. The paper leverages the first few low-degree Hermite polynomials (specifically, up to degree 2) to approximate ReLU.
- Basis-wise Normalization: To mitigate the accuracy degradation often seen with polynomial approximations, AESPA incorporates a novel basis-wise normalization technique. This method normalizes each Hermite polynomial basis independently, ensuring that intermediate values remain stable and preventing issues like vanishing or exploding gradients, which are common with polynomial activations. This normalization is applied after the Hermite expansion and before the weighted summation of the basis functions.
AESPA is designed to directly replace ReLU and batch normalization (BN) layers in existing neural network architectures. The authors demonstrate its application to popular models like VGGNet, ResNet, and pre-activation ResNet.
Datasets
The paper uses the following datasets for training and evaluation:
- CIFAR-10/100: Standard image classification datasets with 32x32 images and 10 or 100 classes, respectively.
- Tiny ImageNet: A larger dataset with 64x64 images and 200 classes.
The models are trained using standard techniques like SGD with cosine annealing, random crops, flips, rotations, and normalization.
Key Insights and Ideas
- Hermite Polynomials as a Superior Choice: The paper suggests that Hermite polynomials, particularly low-degree ones, offer a better trade-off for approximating ReLU in private inference compared to other polynomial approximation methods. Their mathematical properties seem to lend themselves well to preserving accuracy.
- Basis-wise Normalization is Crucial: The proposed basis-wise normalization is identified as a critical component for maintaining high accuracy. It effectively addresses the instability and potential for exploding/vanishing gradients that can plague polynomial activations, especially in deep networks.
- Enabling Full BT Adoption: By achieving comparable accuracy with low-degree polynomials, AESPA allows for the complete replacement of GC-processed ReLUs with BT-processed polynomial activations. This is a significant step towards leveraging the much faster BT primitive for all activation functions.
- Practicality in Real-World PI: The work demonstrates the effectiveness of AESPA not just theoretically but also in a practical PI setting using the Delphi protocol on real machines and networks, showcasing substantial performance gains.
Key Results
- Significant Speedup: When applied to the Delphi PI protocol, AESPA reduces online latency by up to 42.1x and communication costs by up to 28.3x compared to the baseline ReLU-based models.
- Accuracy Preservation: AESPA achieves classification accuracies comparable to standard ReLU-based models on CIFAR-10/100 and Tiny ImageNet. For instance, VGG16 on CIFAR-100 with AESPA shows a 71.99% accuracy, very close to the baseline's 73.45%.
- Outperforming Prior Work: AESPA demonstrates superior accuracy compared to prior low-degree polynomial activation studies (e.g., QuaIL) and competitive accuracy with higher-degree approximations while offering significantly lower latency. For VGG16 on CIFAR-10, AESPA achieves 92.4% accuracy with a 24.0x speedup, outperforming Delphi (88.1% acc, 1.2x speedup), SAFENet (88.9% acc, 2.5x speedup), and Circa (93.8% acc, 2.6x speedup) in terms of speedup.
Limitations and Future Work
- ResNet32 Accuracy Drop: The paper notes a larger accuracy drop for ResNet32 on CIFAR-100 and Tiny ImageNet. This is attributed to the skip-connection in ResNet, which AESPA cannot natively support as directly as it can for models like PA-ResNet. However, the accuracy of the HerPN-based PA-ResNet32 is still competitive.
- Generalizability to Other Activations: While the paper mentions that AESPA can be applied to other activation functions (ELU, SELU, Swish, GeLU, Mish), this is not extensively demonstrated.
- Future Work: The authors suggest exploring the application of AESPA to other PI protocols and investigating its use with a wider range of activation functions. Further optimization of the Hermite polynomial degree and normalization techniques could also be explored.
Summary updated