Abstract
Distributed machine learning (DML) can realize massive dataset training when no single node can work out the accurate results within an acceptable time. However, this will inevitably expose more potential targets to attackers compared with the non-distributed environment. In this paper, we classify DML into basic-DML and semi-DML. In basic-DML, the center server dispatches learning tasks to distributed machines and aggregates their learning results. While in semi-DML, the center server further devotes resources into dataset learning in addition to its duty in basic-DML. We firstly put forward a novel data poison detection scheme for basic-DML, which utilizes a cross-learning mechanism to find out the poisoned data. We prove that the proposed cross-learning mechanism would generate training loops, based on which a mathematical model is established to find the optimal number of training loops. Then, for semi-DML, we present an improved data poison detection scheme to provide better learning protection with the aid of the central resource. To efficiently utilize the system resources, an optimal resource allocation approach is developed. Simulation results show that the proposed scheme can significantly improve the accuracy of the final model by up to 20% for support vector machine and 60% for logistic regression in the basic-DML scenario. Moreover, in the semi-DML scenario, the improved data poison detection scheme with optimal resource allocation can decrease the wasted resources for 20-100%.
1. Introduction
Distributed Machine Learning (DML) has become a cornerstone of modern artificial intelligence, enabling the training of massive models across hundreds or thousands of nodes. However, this distributed architecture significantly increases the "attack surface." If an attacker compromises even a few workers and "poisons" their local data, they can steer the final model toward incorrect or malicious decisions. This paper addresses the challenge of creating a general-purpose detection scheme that works even when the central server has limited resources and cannot verify every single piece of data itself.
2. System Models and Detection Schemes
2.1 Basic-DML vs. Semi-DML
The researchers categorize DML into two primary architectures: Basic-DML and Semi-DML. In Basic-DML, the central server's role is primarily limited to dispatching tasks and aggregating results. In Semi-DML, the center possesses spare computational resources, allowing it to participate in the learning process or verify worker results to enhance security.
2.2 Cross-Learning and Training Loops
To detect poisoned data in Basic-DML, a cross-learning mechanism is introduced. Instead of sending a data chunk to just one worker, it is assigned to multiple workers. This redundancy creates a "virtual topology" of training loops. If workers receiving the same data return significantly different parameters, a potential threat is identified. Using graph theory, the authors established a mathematical model for the Probability of Finding Threats (PFT), proving that increasing the number of these loops makes it significantly harder for attackers to remain undetected.
2.3 Parameter Thresholding and Resource Allocation
To identify abnormal results, the center establishes a "threshold of parameters" by training a small, clean subset of data. Worker results deviating beyond this threshold (measured via Euclidean distance) are flagged. In Semi-DML scenarios, an optimal resource allocation approach is developed to decide whether the center should train data from scratch or verify suspicious results, thereby minimizing wasted computational effort.
3. Performance Evaluation
The proposed schemes were validated using synthetic data for Support Vector Machines (SVM) and the MNIST dataset for Logistic Regression (LR). Key findings include:
- Accuracy Improvement: In Basic-DML, the detection scheme improved final model accuracy by up to 20% for SVM and 60% for Logistic Regression compared to unprotected systems.
- Resource Efficiency: In Semi-DML, the optimal resource allocation approach reduced wasted computational effort by 20% to 100%.
- Model Robustness: The PFT model accurately predicted attack success, confirming that a higher number of training loops provides the most robust protection.
4. Conclusion
This research demonstrates that redundancy, when strategically applied through cross-learning and training loops, serves as a powerful security feature in DML. By optimizing central resource allocation, systems can achieve high accuracy even in compromised environments. Future work will explore dynamic attack patterns and the application of these mechanisms to more complex deep learning architectures like Transformers.
Summary updated