Abstract
Support vector machine (SVM) is one of the most popular machine learning algorithms. It predicts a pre-defined output variable in real-world applications. Machine learning on encrypted data is becoming more and more important to protect both model information and data against various adversaries. While some studies have been proposed on inference or prediction phases, few have been reported on the training phase. Homomorphic encryption (HE) for the arithmetic of approximate numbers scheme enables efficient arithmetic evaluations of encrypted data of real numbers, which encourages to develop privacy-preserving machine learning training algorithm. In this study, we propose an HE-friendly algorithm for the SVM training phase which avoids inefficient operations and numerical instability on an encrypted domain. The inference phase is also implemented on the encrypted domain with fully-homomorphic encryption which enables real-time prediction. Our experiment showed that our HE-friendly algorithm outperformed the state-of-the-art logistic regression classifier with fully homomorphic encryption on toy and real-world datasets. To the best of our knowledge, this study is the first practical algorithm for training an SVM model with fully homomorphic encryption. Therefore, our result supports the development of practical applications of the privacy-preserving SVM model.
Introduction
In the era of stringent data protection regulations like GDPR and the increasing sensitivity of medical and financial analytics, data privacy has transitioned from a desirable feature to a legal and ethical necessity. While significant progress has been made in performing inference—making predictions using pre-trained models—on encrypted data, the task of training a model from scratch without ever decrypting the raw data remains a significant challenge in the field of cryptography.
A recent study published in IEEE Access by Saerom Park and colleagues introduces a practical breakthrough: an efficient algorithm for training Support Vector Machines (SVMs) using Fully Homomorphic Encryption (FHE). This approach allows for the development of robust machine learning models while maintaining complete data confidentiality throughout the training lifecycle.
The Challenge of Encrypted SVM Training
Homomorphic Encryption (HE) allows computations to be performed on ciphertexts such that the decrypted result matches the outcome of the same operations performed on plaintext. However, standard SVM algorithms are inherently "HE-unfriendly." They typically rely on the hinge loss function, which involves a max(0, x) operation, and utilize inequality constraints during optimization.
In the context of HE, non-polynomial functions like max or absolute values are computationally expensive or impossible to implement directly. Furthermore, many HE schemes are designed for exact integer arithmetic, which struggles with the numerical stability and real-number precision required for effective machine learning training.
Proposed Methodology
The researchers addressed these hurdles by redesigning the SVM training process to align with the strengths of modern HE schemes. Their approach relies on three core strategies:
1. Least Squares SVM (LS-SVM) Formulation
Instead of solving the traditional quadratic programming problem with inequality constraints, the authors adopted the Least Squares SVM (LS-SVM) formulation. This modification transforms the training process into solving a system of linear equations. Crucially, this requires only addition and multiplication—the fundamental operations supported efficiently by HE.
2. The HEAAN Scheme
The algorithm utilizes the HEAAN (Homomorphic Encryption for Arithmetic of Approximate Numbers) scheme. Unlike traditional schemes, HEAAN is specifically designed for approximate calculations with real numbers. It treats the minor errors inherent in HE as part of the numerical approximation process, significantly enhancing efficiency for machine learning tasks.
3. Advanced Data Packing and Parallelization
To manage large matrices within the encrypted domain, the authors developed specialized packing strategies. These methods are essential for optimizing memory usage and computational speed.
Two primary strategies were implemented:
- Column-wise packing: This method is highly efficient for small to medium-sized datasets.
- Sub-matrix packing: A scalable approach that enables parallelized matrix multiplication across multiple CPU threads using a modified version of Fox’s algorithm.
To ensure numerical stability against the approximation noise introduced by HEAAN, the training is performed via iterative gradient descent rather than direct matrix inversion.
Experimental Evaluation
The proposed HE-friendly SVM was evaluated against a diverse set of benchmarks, including:
- Synthetic Datasets: A "Linear" dataset and a "Ring" dataset, the latter specifically designed to test the model's ability to handle non-linear classification boundaries.
- Real-World Datasets: Twelve datasets from the UCI Machine Learning Repository, including high-stakes applications such as Wisconsin Breast Cancer diagnosis, German Credit scoring, and Ionosphere data analysis.
Key Results and Insights
The experimental results demonstrate that the HE-friendly SVM is both accurate and efficient:
- Superior Accuracy: On the non-linear "Ring" dataset, the encrypted SVM significantly outperformed state-of-the-art encrypted Logistic Regression (LR) models, which often struggle with complex decision boundaries.
- Enhanced Efficiency: The average time per training iteration for the SVM model was approximately twice as fast as the comparison LR model.
- Real-World Efficacy: On the Wisconsin Breast Cancer dataset, the encrypted SVM achieved an accuracy of 98%, matching the performance of models trained on unencrypted plaintext.
- Scalability: The sub-matrix packing method proved superior for larger datasets, allowing the algorithm to scale effectively by leveraging additional computational threads.
Conclusion
This work represents a major step toward practical privacy-preserving machine learning. By selecting mathematical variants like LS-SVM that naturally align with cryptographic primitives, the authors have shown that training on encrypted data is not only possible but increasingly feasible for real-world applications.
While training still takes longer than plaintext operations (seconds per iteration versus milliseconds), the gap is closing. Future work will likely focus on extending these methods to high-dimensional multi-class problems and automating hyperparameter tuning, such as kernel selection, entirely within the encrypted domain.
Summary updated