Abstract
Nowadays, gathering high-quality training data from multiple data sources with privacy preservation is a crucial challenge to training high-performance machine learning models. The potential solutions could break the barriers among isolated data corpus, and consequently enlarge the range of data available for processing. To this end, both academic researchers and industrial vendors are recently strongly motivated to propose two main-stream folders of solutions mainly based on software constructions: 1) Secure Multi-party Learning (MPL for short); and 2) Federated Learning (FL for short). The above two technical folders have their advantages and limitations when we evaluate them according to the following five criteria: security, efficiency, data distribution, the accuracy of trained models, and application scenarios. Motivated to demonstrate the research progress and discuss the insights on the future directions, we thoroughly investigate these protocols and frameworks of both MPL and FL. At first, we define the problem of Training machine learning Models over Multiple data sources with Privacy Preservation (TMMPP for short). Then, we compare the recent studies of TMMPP from the aspects of the technical routes, the number of parties supported, data partitioning, threat model, and machine learning models supported, to show their advantages and limitations. Next, we investigate and evaluate five popular FL platforms. Finally, we discuss the potential directions to resolve the problem of TMMPP in the future.
Introduction
As data privacy regulations like GDPR tighten, the "isolated data island" problem—where valuable data is trapped in silos—has become a major bottleneck for AI. This Systematization of Knowledge (SoK) paper provides a comprehensive roadmap for Privacy-Preserving Machine Learning (PPML), a field becoming the backbone of modern data science. Researchers from Fudan University categorize and benchmark the two primary solutions to this problem: Secure Multi-party Learning (MPL) and Federated Learning (FL).
Defining the TMMPP Problem
The authors define the core challenge as TMMPP: how can multiple parties jointly train a high-performance model without ever disclosing their raw data to each other or a third party? This process is hindered by three fundamental hurdles that any viable solution must address.
- Security Challenge: Preventing "honest-but-curious" or malicious parties from inferring private data from intermediate model updates.
- Efficiency Challenge: Overcoming the massive communication and computation overhead caused by cryptographic protocols.
- Statistical Challenge: Dealing with "Non-IID" data (where data distributions vary wildly between participants), which can cause model accuracy to plummet.
Technical Methodologies: MPL vs. FL
The technical landscape is divided into two primary architectural approaches:
Secure Multi-party Learning (MPL)
MPL typically utilizes a Peer-to-Peer (P2P) architecture and relies on heavy-duty cryptography. Key techniques include Homomorphic Encryption (HE), which allows computing on encrypted data; Garbled Circuits (GC), which represent functions as logic gates to hide inputs; and Secret Sharing (SS), which splits data into "shares" distributed among parties.
MPL is notably "Data-Blind." Unlike FL, it is generally not sensitive to how data is distributed (IID vs. non-IID) because data is cryptographically shared before training starts. However, it faces a "Scalability Wall"; most frameworks are tailored for only 2 or 3 parties, as communication overhead explodes with more participants.
Federated Learning (FL)
FL typically uses a Client-Server architecture where parties train models locally and only share model updates (gradients) with a central server. To protect these updates, FL often incorporates Differential Privacy (DP) to mask individual contributions and Secure Aggregation to ensure the server only sees the aggregate sum of updates.
Benchmarking and Insights
The authors evaluated five major open-source platforms—FATE, TFF (TensorFlow Federated), PaddleFL, PySyft, and coMind—using the standard MNIST dataset to measure the performance trade-offs of privacy layers.
- Platform Performance: FATE and PaddleFL emerged as the most robust for industrial use, supporting both horizontal (different users, same features) and vertical (same users, different features) data partitioning. Conversely, PySyft was found to be significantly slower due to its "command replication" architecture.
- Accuracy and Scalability: In FL, as the number of participants increases, model accuracy generally trends downward due to data heterogeneity. For most platforms, there is an "efficiency sweet spot" where training time initially decreases due to parallelism but eventually increases as the overhead of managing many connections takes over.
Future Directions and Limitations
Despite rapid progress, several challenges remain for future research:
- Model Complexity: Current frameworks are optimized for simple models like Logistic Regression; supporting complex architectures like LSTMs or Decision Trees remains inefficient.
- System Heterogeneity: Real-world networks often have participants with varying hardware and unstable connections, which can bottleneck the entire training process.
- Malicious Actors: Protecting against a truly malicious central server in FL and handling "dynamic" membership (where parties drop out mid-training) are critical areas for future development.
Summary updated