TL;DR
This paper tackles critical accuracy and efficiency issues in privacy-preserving machine learning (PPML) caused by probabilistic truncation protocols. The authors introduce a novel deterministic truncation technique and optimize ReLU/DReLU evaluations, leading to significant speedups (3-4x overall) without compromising model accuracy.
What Problem Does This Paper Address?
Privacy-preserving machine learning (PPML) aims to train and infer models without revealing sensitive data or parameters. A common challenge in PPML, particularly in multi-party computation (MPC) frameworks, is the need to handle fixed-point arithmetic and rescale data ranges. This often involves "truncation protocols." Many existing PPML platforms rely on probabilistic truncation, which, while offering advantages like reduced communication, suffers from inherent errors (denoted as $e_1$) that can severely degrade model accuracy. Furthermore, these protocols often require large computational rings (e.g., 64-bit) to mitigate these errors, impacting efficiency. This paper addresses these dual problems of accuracy degradation and efficiency bottlenecks stemming from probabilistic truncation in PPML.
Methods
The core of this work lies in two main innovations:
- Deterministic Truncation: The authors propose a new non-interactive deterministic truncation technique to replace probabilistic methods. This technique is built upon a formal analysis of "cut functions" and their properties, directly addressing the error ($e_1$) that plagues probabilistic approaches. They also introduce a more general deterministic truncation that can cut both leading and trailing bits.
- Optimized ReLU/DReLU Evaluation: Recognizing that non-linear functions like ReLU are computationally expensive in PPML, the paper introduces a framework to optimize their evaluation. This involves a "post-training calibration" step to determine a scaling factor ($s$) for the input to ReLU. This scaled input is then processed using their deterministic truncation and a modified DReLU evaluation. This approach effectively reduces the precision required for intermediate computations, leading to significant communication and computation savings. They also propose an improved DReLU protocol that uses a zero-preserving random mapping to ensure security and correctness.
Datasets
Not discussed in the paper. The paper focuses on theoretical analysis and experimental evaluation using existing PPML platforms and models, rather than introducing new datasets.
Key Insights and Ideas
- Probabilistic Truncation's Hidden Flaw: The paper reveals that common probabilistic truncation protocols, while seemingly robust, are prone to a specific error ($e_1$) that can cause significant accuracy drops, especially when the precision of the input ($l_x$) is close to the ring size ($l$).
- The "Slack" Problem: The need for a "slack" (i.e., $l \gg l_x$) to minimize $e_1$ in probabilistic protocols is a major efficiency bottleneck, forcing the use of larger, more communication-heavy rings.
- Deterministic is Better (for this): By moving to a deterministic truncation, the authors eliminate the primary source of accuracy degradation ($e_1$), allowing for smaller ring sizes and thus improved efficiency.
- Scaling for Efficiency: The insight that many non-linear functions (like ReLU) can operate on scaled inputs without affecting overall model accuracy is key. This allows for reducing the precision of intermediate computations, directly translating to performance gains.
- Implementation Bugs Matter: The paper highlights how some existing works might have masked the $e_1$ error by using fixed random numbers instead of truly random ones, leading to misleading accuracy results.
Key Results
- Accuracy Preservation: The proposed deterministic truncation and optimized ReLU/DReLU evaluation achieve model accuracy comparable to original methods, with negligible loss (e.g., ~0.2% for 7-bit ReLU).
- Significant Speedups:
- The optimized ReLU protocol achieves over 10x speedup compared to Piranha-Falcon and over 3x improvement over Bicoptor.
- The overall PPML model inference can be sped up by 3-4 times.
- A 7-bit DReLU evaluation can be sped up approximately 5x compared to ABY³, ABY 2.0, EdaBits, and Bicoptor.
- Communication Reduction:
- The improved Bicoptor ReLU protocol reduces communication by 92%.
- The optimized protocol can perform a ReLU evaluation within 2 rounds and 704 bits of communication (for 64-bit ring and 7-bit precision).
Limitations and Future Work
- Limitations:
- The paper acknowledges that while their deterministic truncation is non-interactive, some PPML protocols might still require interactive elements for other operations.
- The "post-training calibration" step for determining the scaling factor ($s$) adds a calibration phase, which needs to be performed on a dataset representative of the training data.
- Future Work:
- The authors suggest that their framework can be extended to other non-linear functions beyond ReLU and Maxpool.
- Further exploration of optimizing other computationally intensive parts of PPML inference is implied.
- Investigating the security and efficiency trade-offs of their deterministic truncation in different MPC settings could be a future direction.
Summary updated