Neural Networks with (Low-Precision) Polynomial Approximations: New Insights and Techniques for Accuracy Improvement

Chi Zhang, Man Ho Au, Siu Ming YiuarXiv 2024 · arXiv · OpenReview

Abstract

Replacing non-polynomial functions (e.g., non-linear activation functions such as ReLU) in a neural network with their polynomial approximations is a standard practice in privacy-preserving machine learning. The resulting neural network, called polynomial approximation of neural network (PANN) in this paper, is compatible with advanced cryptosystems to enable privacy-preserving model inference. Using "highly precise" approximation, state-of-the-art PANN offers similar inference accuracy as the underlying backbone model. However, little is known about the effect of approximation, and existing literature often determined the required approximation precision empirically. In this paper, we initiate the investigation of PANN as a standalone object. Specifically, our contribution is two-fold. Firstly, we provide an explanation on the effect of approximate error in PANN. In particular, we discovered that (1) PANN is susceptible to some type of perturbations; and (2) weight regularisation significantly reduces PANN's accuracy. We support our explanation with experiments. Secondly, based on the insights from our investigations, we propose solutions to increase inference accuracy for PANN. Experiments showed that combination of our solutions is very effective: at the same precision, our PANN is 10% to 50% more accurate than state-of-the-arts; and at the same accuracy, our PANN only requires a precision of $2^{-9}$ while state-of-the-art solution requires a precision of $2^{-12}$ using the ResNet-20 model on CIFAR-10 dataset.

Introduction

Privacy-preserving machine learning (PPML) leverages cryptographic techniques like Multi-Party Computation (MPC) or Homomorphic Encryption (HE) to perform inference on sensitive data. A significant challenge in this domain is that these cryptosystems natively support only arithmetic operations like addition and multiplication. Standard activation functions like ReLU ($max(0, x)$) must therefore be replaced with polynomial approximations to create Polynomial Approximation of Neural Networks (PANNs).

While high-precision polynomials can maintain accuracy, they are computationally expensive in encrypted environments. Conversely, low-precision approximations are fast but often lead to severe accuracy degradation. This paper investigates the "sturdiness" of models—their inherent resistance to these approximation errors—and proposes techniques to improve it without sacrificing speed.

The Sensitivity of Negative Inputs

A key discovery in this research is that PANNs are disproportionately sensitive to approximation errors occurring on the negative side of the ReLU function. In a standard ReLU, negative inputs result in a zero output with a zero gradient. However, polynomial approximations introduce "leakage" or noise in this region, which disrupts the stationary points found during training.

Figure 1: Loss sensitivity to positive (pos) vs. negative (neg) input perturbations across different architectures. The loss increases significantly more for negative perturbations, especially when weight decay (wd) is present.

As shown in Figure 1, across various architectures like ResNet-20 and MobileNetv2, perturbations on negative inputs (neg) lead to a much sharper increase in loss compared to positive inputs (pos). To address this, the authors propose the Noise Generator for Negative Values (NGNV). This technique injects simulated noise specifically into negative inputs during training, effectively teaching the model to be robust where it is most vulnerable.

Strategic Regularization and Weight Decay

Standard training practices, particularly weight decay, were found to be detrimental to PANN sturdiness. Mathematical analysis reveals that weight decay increases gradient norms in a way that amplifies the impact of approximation noise. This creates a paradox where a technique meant to improve generalization actually tanks the performance of the approximated model.

Figure 2: Accuracy over training epochs for backbone (bb) models and PANNs with varying weight decay (wd) values. Higher weight decay significantly degrades PANN accuracy compared to the backbone.

Figure 2 illustrates this phenomenon: while the backbone models (dotted lines) maintain stability, the PANNs (solid lines) show a dramatic drop in accuracy as weight decay increases (e.g., $wd=5e-4$). To maintain generalization without the negative side effects of weight decay, the authors suggest reducing weight decay and utilizing Mixup training (training on convex combinations of image pairs) instead.

Results and Efficiency Gains

By combining NGNV and strategic regularization, the researchers achieved significant improvements in both accuracy and efficiency. On the CIFAR-10 dataset with ResNet-20, their "sturdy" training allowed for high accuracy at a precision of $2^{-9}$. In practical terms, this lower precision requirement reduced the inference time from approximately 71.2 seconds (at $2^{-12}$ precision) to 33.6 seconds—a speedup of over 50%.

Overall, the proposed PANNs were 10% to 50% more accurate than existing state-of-the-art solutions at equivalent precision levels. These findings provide a theoretical and empirical foundation for building faster, more robust privacy-preserving AI systems.

Summary updated

Abstract

Replacing non-polynomial functions (e.g., non-linear activation functions such as ReLU) in a neural network with their polynomial approximations is a standard practice in privacy-preserving machine learning. The resulting neural network, called polynomial approximation of neural network (PANN) in this paper, is compatible with advanced cryptosystems to enable privacy-preserving model inference. Using ``highly precise'' approximation, state-of-the-art PANN offers similar inference accuracy as the underlying backbone model. However, little is known about the effect of approximation, and existing literature often determined the required approximation precision empirically. In this paper, we initiate the investigation of PANN as a standalone object. Specifically, our contribution is two-fold. Firstly, we provide an explanation on the effect of approximate error in PANN. In particular, we discovered that (1) PANN is susceptible to some type of perturbations; and (2) weight regularisation significantly reduces PANN's accuracy. We support our explanation with experiments. Secondly, based on the insights from our investigations, we propose solutions to increase inference accuracy for PANN. Experiments showed that combination of our solutions is very effective: at the same precision, our PANN is 10% to 50% more accurate than state-of-the-arts; and at the same accuracy, our PANN only requires a precision of 2^{-9} while state-of-the-art solution requires a precision of 2^{-12} using the ResNet-20 model on CIFAR-10 dataset.
BibTeX
@misc{zhang2024neural,
  author = {Chi Zhang and Man Ho Au and Siu Ming Yiu},
  title = {{Neural Networks with (Low-Precision) Polynomial Approximations: New Insights and Techniques for Accuracy Improvement}},
  year = {2024},
  eprint = {2402.11224},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2402.11224},
}
Figure