Multi-Party Dual Learning

Maoguo Gong, Yuan Gao, Yu Xie, A. Kai Qin, Ke Pan, Yew-Soon OngarXiv 2021 · OpenReview

TL;DR

This paper introduces Multi-Party Dual Learning (MPDL), a novel framework for distributed machine learning that tackles the challenge of limited and non-overlapping data across different parties. By leveraging dual learning principles and a feature-oriented differential privacy mechanism, MPDL enables robust model training while preserving data privacy, outperforming existing multi-party learning methods.

What Problem Does This Paper Address?

In today's data-driven world, machine learning models thrive on vast amounts of data. However, real-world data is often fragmented across various institutions, each with its own privacy policies and data regulations (like GDPR and CCPA). This makes it nearly impossible to pool data for centralized training. Consequently, individual parties might have insufficient or low-quality data, hindering their ability to build effective models. Existing multi-party learning approaches often struggle when data samples or features don't overlap significantly, leading to wasted data and suboptimal model performance. This paper aims to bridge this gap by enabling collaborative learning even with minimal data overlap, while strictly adhering to privacy constraints.

Methods

The core of the proposed solution is the Multi-Party Dual Learning (MPDL) framework. It builds upon the concept of dual learning, where two tasks are intrinsically linked and can inform each other. In this multi-party context, each party trains a "dual model" that learns to map data from its own space to the other party's space and vice-versa. This process helps generate "synthetic" overlapping data, effectively expanding the training set.

To address privacy concerns, MPDL introduces feature-oriented differential privacy. This involves an affine transformation layer that perturbs raw features with Laplace noise before they are used in the dual learning process. This perturbation ensures that even if a party could infer information about the other's data, it would be noisy and generalized, preventing the leakage of specific private features. The framework also incorporates additive homomorphic encryption for transmitting encrypted gradients and probability distributions, further safeguarding sensitive information. A collaborator (a trusted third party) is used for aggregating model updates and training a central model, but it only receives perturbed data and encrypted information, not raw data or model structures.

Datasets

The paper evaluates MPDL on several real-world datasets:

  • MNIST: A dataset of handwritten digits.
  • CIFAR-10: A dataset of color images.
  • Bank Marketing: Data related to direct marketing campaigns for a Portuguese banking institution.
  • Breast Cancer Wisconsin (Diagnostic): Features computed from fine needle aspirates for breast mass diagnosis.
  • Google+ dataset: An ego-network of Google+ users.
  • Hamilton dataset: A collection of US university Facebook networks.

These datasets are split to simulate a two-party distributed learning scenario, with varying degrees of data overlap.

Key Insights and Ideas

  • Dual learning is a natural fit for fragmented data: The inherent "dual" nature of knowledge sharing between parties makes dual learning a suitable paradigm for scenarios with limited overlapping data.
  • Synthetic data generation bridges the gap: By learning to map between data spaces, dual models can generate valuable synthetic overlapping samples, significantly augmenting the training data for the central model.
  • Feature-level privacy is crucial: Perturbing raw features before any cross-party interaction is key to preventing specific private information leakage, even when data is not perfectly aligned.
  • Privacy and accuracy can be balanced: The framework demonstrates that by tuning the privacy budget (epsilon), a trade-off can be managed, allowing for strong privacy guarantees without completely sacrificing model performance.
  • Minimal disruption to existing structures: MPDL requires minimal modifications to existing multi-party learning setups, making it more adaptable.

Key Results

  • Significant accuracy improvements: MPDL consistently outperforms baseline methods like "jointy" (training on available overlapping data) and Federated Transfer Learning (FTL) across various datasets and co-occurrence probabilities. For instance, on MNIST with low co-occurrence probability, MPDLA achieved accuracy improvements of up to 10% over "jointy."
  • Robustness with limited overlap: The framework shows strong performance even when the overlap between parties is as low as 5%.
  • Effective privacy preservation: Experiments demonstrate that increasing the privacy budget (epsilon) leads to higher accuracy but also higher inference error (indicating less privacy), showcasing the ability to tune this trade-off.
  • Comparable to centralized learning: In some cases, MPDL achieves accuracy comparable to a non-distributed model trained on fully gathered data, highlighting its effectiveness.
  • Qualitative improvements: Visualizations show that MPDL preserves the intrinsic structure of data better than FTL, especially in scenarios with damaged or missing parts of images.

Limitations and Future Work

  • Computational complexity: While the framework is designed to be efficient, the dual learning process and differential privacy mechanisms can introduce computational overhead.
  • Third-party collaborator reliance: The current MPDL relies on a trusted third-party collaborator for central model training. Reducing this dependency is a future research direction.
  • Scalability to more complex models: The paper validates MPDL with MLP and CNN models. Future work aims to investigate its scalability with more complex architectures like recurrent neural networks (RNNs) and graph neural networks (GNNs).
  • Exploring new privacy schemes: The authors suggest exploring new schemes to further reduce model complexity and dependency on the collaborator, potentially leading to improved utility and communication efficiency.
  • Application to diverse tasks: While tested on image and graph data, extending MPDL to other domains like text processing is a promising avenue.

Summary updated

Abstract

The performance of machine learning algorithms heavily relies on the availability of a large amount of training data. However, in reality, data usually reside in distributed parties such as different institutions and may not be directly gathered and integrated due to various data policy constraints. As a result, some parties may suffer from insufficient data available for training machine learning models. In this paper, we propose a multi-party dual learning (MPDL) framework to alleviate the problem of limited data with poor quality in an isolated party. Since the knowledge sharing processes for multiple parties always emerge in dual forms, we show that dual learning is naturally suitable to handle the challenge of missing data, and explicitly exploits the probabilistic correlation and structural relationship between dual tasks to regularize the training process. We introduce a feature-oriented differential privacy with mathematical proof, in order to avoid possible privacy leakage of raw features in the dual inference process. The approach requires minimal modifications to the existing multi-party learning structure, and each party can build flexible and powerful models separately, whose accuracy is no less than non-distributed self-learning approaches. The MPDL framework achieves significant improvement compared with state-of-the-art multi-party learning methods, as we demonstrated through simulations on real-world datasets.
BibTeX
@misc{gong2021multi,
  author = {Maoguo Gong and Yuan Gao and Yu Xie and A. Kai Qin and Ke Pan and Yew-Soon Ong},
  title = {{Multi-Party Dual Learning}},
  year = {2021},
  eprint = {2104.06677},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2104.06677},
}
Figure