TL;DR
This paper introduces Multi-Party Dual Learning (MPDL), a novel framework for distributed machine learning that tackles the challenge of limited and non-overlapping data across different parties. By leveraging dual learning principles and a feature-oriented differential privacy mechanism, MPDL enables robust model training while preserving data privacy, outperforming existing multi-party learning methods.
What Problem Does This Paper Address?
In today's data-driven world, machine learning models thrive on vast amounts of data. However, real-world data is often fragmented across various institutions, each with its own privacy policies and data regulations (like GDPR and CCPA). This makes it nearly impossible to pool data for centralized training. Consequently, individual parties might have insufficient or low-quality data, hindering their ability to build effective models. Existing multi-party learning approaches often struggle when data samples or features don't overlap significantly, leading to wasted data and suboptimal model performance. This paper aims to bridge this gap by enabling collaborative learning even with minimal data overlap, while strictly adhering to privacy constraints.
Methods
The core of the proposed solution is the Multi-Party Dual Learning (MPDL) framework. It builds upon the concept of dual learning, where two tasks are intrinsically linked and can inform each other. In this multi-party context, each party trains a "dual model" that learns to map data from its own space to the other party's space and vice-versa. This process helps generate "synthetic" overlapping data, effectively expanding the training set.
To address privacy concerns, MPDL introduces feature-oriented differential privacy. This involves an affine transformation layer that perturbs raw features with Laplace noise before they are used in the dual learning process. This perturbation ensures that even if a party could infer information about the other's data, it would be noisy and generalized, preventing the leakage of specific private features. The framework also incorporates additive homomorphic encryption for transmitting encrypted gradients and probability distributions, further safeguarding sensitive information. A collaborator (a trusted third party) is used for aggregating model updates and training a central model, but it only receives perturbed data and encrypted information, not raw data or model structures.
Datasets
The paper evaluates MPDL on several real-world datasets:
- MNIST: A dataset of handwritten digits.
- CIFAR-10: A dataset of color images.
- Bank Marketing: Data related to direct marketing campaigns for a Portuguese banking institution.
- Breast Cancer Wisconsin (Diagnostic): Features computed from fine needle aspirates for breast mass diagnosis.
- Google+ dataset: An ego-network of Google+ users.
- Hamilton dataset: A collection of US university Facebook networks.
These datasets are split to simulate a two-party distributed learning scenario, with varying degrees of data overlap.
Key Insights and Ideas
- Dual learning is a natural fit for fragmented data: The inherent "dual" nature of knowledge sharing between parties makes dual learning a suitable paradigm for scenarios with limited overlapping data.
- Synthetic data generation bridges the gap: By learning to map between data spaces, dual models can generate valuable synthetic overlapping samples, significantly augmenting the training data for the central model.
- Feature-level privacy is crucial: Perturbing raw features before any cross-party interaction is key to preventing specific private information leakage, even when data is not perfectly aligned.
- Privacy and accuracy can be balanced: The framework demonstrates that by tuning the privacy budget (epsilon), a trade-off can be managed, allowing for strong privacy guarantees without completely sacrificing model performance.
- Minimal disruption to existing structures: MPDL requires minimal modifications to existing multi-party learning setups, making it more adaptable.
Key Results
- Significant accuracy improvements: MPDL consistently outperforms baseline methods like "jointy" (training on available overlapping data) and Federated Transfer Learning (FTL) across various datasets and co-occurrence probabilities. For instance, on MNIST with low co-occurrence probability, MPDLA achieved accuracy improvements of up to 10% over "jointy."
- Robustness with limited overlap: The framework shows strong performance even when the overlap between parties is as low as 5%.
- Effective privacy preservation: Experiments demonstrate that increasing the privacy budget (epsilon) leads to higher accuracy but also higher inference error (indicating less privacy), showcasing the ability to tune this trade-off.
- Comparable to centralized learning: In some cases, MPDL achieves accuracy comparable to a non-distributed model trained on fully gathered data, highlighting its effectiveness.
- Qualitative improvements: Visualizations show that MPDL preserves the intrinsic structure of data better than FTL, especially in scenarios with damaged or missing parts of images.
Limitations and Future Work
- Computational complexity: While the framework is designed to be efficient, the dual learning process and differential privacy mechanisms can introduce computational overhead.
- Third-party collaborator reliance: The current MPDL relies on a trusted third-party collaborator for central model training. Reducing this dependency is a future research direction.
- Scalability to more complex models: The paper validates MPDL with MLP and CNN models. Future work aims to investigate its scalability with more complex architectures like recurrent neural networks (RNNs) and graph neural networks (GNNs).
- Exploring new privacy schemes: The authors suggest exploring new schemes to further reduce model complexity and dependency on the collaborator, potentially leading to improved utility and communication efficiency.
- Application to diverse tasks: While tested on image and graph data, extending MPDL to other domains like text processing is a promising avenue.
Summary updated