Towards Fair and Decentralized Privacy-Preserving Deep Learning with Blockchain

Lingjuan Lyu, Jiangshan Yu, Karthik Nandakumar, Yitong Li, Xingjun Ma, Jiong JinarXiv 2019 · arXiv

Introduction

Deep learning's success is heavily dependent on the availability of large-scale datasets. However, data is frequently siloed within organizations—such as financial institutions or healthcare providers—due to stringent privacy regulations and competitive concerns. While distributed learning techniques like Federated Learning (FL) have emerged to address these silos by allowing joint model training without raw data sharing, they introduce two critical vulnerabilities: a single point of failure via central parameter servers and a lack of "collaborative fairness."

Traditional distributed learning architecture relying on a central parameter server, which creates a single point of failure.

In traditional frameworks, all participants receive the same global model regardless of their contribution. This allows "free-riders"—parties contributing minimal or low-quality data—to benefit equally from the high-quality contributions of others, discouraging participation from major data owners. This paper introduces the Decentralized Privacy-Preserving Deep Learning (DPPDL) framework, which leverages blockchain and a credibility-based economy to ensure that the model accuracy a participant receives is directly proportional to the quality and quantity of their data contribution.

Methods: The DPPDL Framework

The DPPDL framework moves away from centralized architectures toward a decentralized, peer-to-peer (P2P) system where gradients are exchanged directly between participants.

The proposed decentralized P2P gradient exchange architecture within the DPPDL framework.

The framework operates in two distinct stages:

Stage 1: Initial Benchmarking

To mitigate the entry of parasitic participants, DPPDL employs an initial benchmarking phase. Each party trains a Differentially Private GAN (DPGAN) on their local dataset to generate synthetic, privacy-preserving samples. These samples are shared among participants, who then label each other's data. This process establishes an initial "local credibility" score for each party, ensuring that only those with meaningful data can effectively participate in the subsequent training.

Stage 2: Privacy-Preserving Collaborative Training

Training is managed through a blockchain-based "points economy." Participants earn points by uploading high-quality gradients and spend them to download updates from others. This creates a marketplace where model updates are treated as tradable commodities.

To maintain privacy during gradient exchange, DPPDL utilizes a three-layer onion encryption stack:

  1. Functional/Homomorphic Layer: Employs a symmetric-key scheme (Vernam cipher) allowing for the aggregation of gradients without exposing individual updates.
  2. Symmetric Layer: Uses fresh AES keys for efficient data encryption.
  3. Asymmetric Layer: Encrypts the AES keys using the receiver’s RSA public key, ensuring that only authorized parties can access the updates.

Experimental Evaluation

The framework was evaluated using the MNIST and SVHN (Street View House Numbers) datasets. SVHN provides a more rigorous test due to its inherent noise and complexity. The researchers tested the system under various scenarios, including uniform privacy levels, varying privacy budgets, and imbalanced data distributions.

Key Results

  • Collaborative Fairness: The authors measured fairness as the correlation between a party's contribution and their resulting model accuracy. While standard distributed learning often showed negligible fairness, DPPDL achieved correlation coefficients between 0.92 and 0.96 in imbalanced scenarios.
  • Model Performance: DPPDL maintains high utility. On the MNIST dataset with 50 participants, the framework achieved ~98.2% accuracy, nearly matching a centralized non-private model (~98.6%) and significantly outperforming standalone training (~94%).
  • Computational Efficiency: Despite the complex encryption stack, the overhead remains minimal. Encrypting a standard model update (0.56 MB) takes approximately 13 milliseconds per participant.

Discussion and Conclusion

DPPDL successfully addresses the free-rider problem while maintaining data privacy and model utility. By treating model updates as a commodity within a credibility marketplace, the system naturally filters out low-quality contributors.

However, the decentralized nature of the protocol results in a communication complexity of $O(n^2)$, making it most suitable for consortiums of 10-50 large institutions rather than massive networks of edge devices. Future work will focus on hardening the system against more sophisticated malicious attacks and testing the framework in real-world network environments to assess latency impacts.

Summary updated

Abstract

The current standalone deep learning framework tends to result in overfitting and low utility. This problem can be addressed by either a centralized framework that deploys a central server to train a global model on the joint data from all parties, or a distributed framework that leverages a parameter server to aggregate local model updates. Server-based solutions are prone to the problem of a single-point-of-failure. In this respect, collaborative learning frameworks, such as federated learning (FL), are more robust. Existing federated learning frameworks overlook an important aspect of participation: fairness. All parties are given the same final model without regard to their contributions. To address these issues, we propose a decentralized Fair and Privacy-Preserving Deep Learning (FPPDL) framework to incorporate fairness into federated deep learning models. In particular, we design a local credibility mutual evaluation mechanism to guarantee fairness, and a three-layer onion-style encryption scheme to guarantee both accuracy and privacy. Different from existing FL paradigm, under FPPDL, each participant receives a different version of the FL model with performance commensurate with his contributions. Experiments on benchmark datasets demonstrate that FPPDL balances fairness, privacy and accuracy. It enables federated learning ecosystems to detect and isolate low-contribution parties, thereby promoting responsible participation.
BibTeX
@misc{lyu2019towards,
  author = {Lingjuan Lyu and Jiangshan Yu and Karthik Nandakumar and Yitong Li and Xingjun Ma and Jiong Jin},
  title = {{Towards Fair and Decentralized Privacy-Preserving Deep Learning with Blockchain}},
  year = {2019},
  eprint = {1906.01167},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1906.01167},
}
Figure