Can we Generalize and Distribute Private Representation Learning?

Sheikh Shams Azam, Taejin Kim, Seyyedali Hosseinalipour, Carlee Joe-Wong, Saurabh Bagchi, Christopher G. Brinton

Abstract

Private Representation Learning (PRL) seeks to transform raw data into representations that preserve utility for specific tasks while concealing sensitive information. Current PRL methods often struggle with multi-attribute protection and the constraints of decentralized data. We propose the Exclusion-Inclusion Generative Adversarial Network (EIGAN), a framework designed for multi-target PRL, and its distributed counterpart, D-EIGAN. Our experiments demonstrate that EIGAN achieves a 47% accuracy improvement over existing state-of-the-art methods. Furthermore, D-EIGAN enables effective learning across siloed data nodes without raw data transmission, maintaining performance comparable to centralized models.

Introduction

The proliferation of data-driven applications has heightened the need for privacy-preserving machine learning. Private Representation Learning (PRL) addresses this by learning an encoder that filters out sensitive "adversary" attributes while retaining "ally" information. However, existing solutions are often limited to single-attribute scenarios and assume centralized data access. In practice, data is frequently distributed across multiple silos (e.g., healthcare institutions) and contains multiple sensitive dimensions. This paper introduces EIGAN and D-EIGAN to overcome these limitations.

Proposed Framework: EIGAN and D-EIGAN

EIGAN generalizes the PRL problem by modeling it as a minimax game involving an encoder ($E$), multiple ally networks ($A_1, \dots, A_n$), and multiple adversary networks ($V_1, \dots, V_m$). The objective is to maximize the utility for all allies while minimizing the information leakage to all adversaries. A critical innovation in EIGAN is the use of Cross-Entropy (CE) loss, which provides a more stable training signal than KL-divergence and effectively pushes adversary predictions toward a uniform distribution, representing total uncertainty.

Architecture of EIGAN (a) featuring multiple ally and adversary networks, and the distributed D-EIGAN (b) architecture for siloed data.

To address data decentralization, D-EIGAN allows each node to train a local encoder. These nodes periodically synchronize their parameters with a central server. To optimize communication and enhance privacy, D-EIGAN employs sparsification, sharing only a random fraction of model weights. This technique not only reduces bandwidth but also helps mitigate local data biases (the Sattler effect), leading to a more robust global representation.

Evaluation and Results

The performance of EIGAN was evaluated across diverse benchmarks, including MNIST, MIMIC-III, and FaceScrub. On the FaceScrub dataset, EIGAN achieved a 47% improvement in ally prediction accuracy compared to the Bertran-PRL baseline, without increasing sensitive information leakage.

Performance comparison of EIGAN against baselines. (a) Adversary log loss (higher is better for privacy). (b) Ally log loss (lower is better for utility).

As illustrated in the experimental results, EIGAN (red line) consistently outperforms baselines such as standard Autoencoders, PCA, and Differential Privacy (DP) methods. It simultaneously achieves the lowest ally loss and the highest adversary loss relative to non-private baselines. Additionally, D-EIGAN demonstrated consistent performance parity with the centralized EIGAN, proving that distributed learning does not necessitate a trade-off in representation quality.

Conclusion

EIGAN and D-EIGAN represent a significant advancement in Private Representation Learning by providing a scalable, multi-attribute, and distributed framework. By leveraging adversarial training and sparsified parameter synchronization, these models offer a practical path toward privacy-preserving data sharing in complex, decentralized environments. Future research will explore the integration of formal differential privacy guarantees and the optimization of training for high-dimensional attribute sets.

Summary updated

Abstract

We study the problem of learning representations that are private yet informative i.e., provide information about intended "ally" targets while hiding sensitive "adversary" attributes. We propose Exclusion-Inclusion Generative Adversarial Network (EIGAN), a generalized private representation learning (PRL) architecture that accounts for multiple ally and adversary attributes unlike existing PRL solutions. While centrally-aggregated dataset is a prerequisite for most PRL techniques, data in real-world is often siloed across multiple distributed nodes unwilling to share the raw data because of privacy concerns. We address this practical constraint by developing D-EIGAN, the first distributed PRL method that learns representations at each node without transmitting the source data. We theoretically analyze the behavior of adversaries under the optimal EIGAN and D-EIGAN encoders and the impact of dependencies among ally and adversary tasks on the optimization objective. Our experiments on various datasets demonstrate the advantages of EIGAN in terms of performance, robustness, and scalability. In particular, EIGAN outperforms the previous state-of-the-art by a significant accuracy margin ($47%$ improvement), and D-EIGAN’s performance is consistently on par with EIGAN under different network settings.
BibTeX
@inproceedings{azam2022can,
  author = {Sheikh Shams Azam and Taejin Kim and Seyyedali Hosseinalipour and Carlee Joe-Wong and Saurabh Bagchi and Christopher G. Brinton},
  title = {{Can we Generalize and Distribute Private Representation Learning?}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2022},
  url = {https://openreview.net/forum?id=mMeWqc1DyX},
}
Figure