Abstract
Private Representation Learning (PRL) seeks to transform raw data into representations that preserve utility for specific tasks while concealing sensitive information. Current PRL methods often struggle with multi-attribute protection and the constraints of decentralized data. We propose the Exclusion-Inclusion Generative Adversarial Network (EIGAN), a framework designed for multi-target PRL, and its distributed counterpart, D-EIGAN. Our experiments demonstrate that EIGAN achieves a 47% accuracy improvement over existing state-of-the-art methods. Furthermore, D-EIGAN enables effective learning across siloed data nodes without raw data transmission, maintaining performance comparable to centralized models.
Introduction
The proliferation of data-driven applications has heightened the need for privacy-preserving machine learning. Private Representation Learning (PRL) addresses this by learning an encoder that filters out sensitive "adversary" attributes while retaining "ally" information. However, existing solutions are often limited to single-attribute scenarios and assume centralized data access. In practice, data is frequently distributed across multiple silos (e.g., healthcare institutions) and contains multiple sensitive dimensions. This paper introduces EIGAN and D-EIGAN to overcome these limitations.
Proposed Framework: EIGAN and D-EIGAN
EIGAN generalizes the PRL problem by modeling it as a minimax game involving an encoder ($E$), multiple ally networks ($A_1, \dots, A_n$), and multiple adversary networks ($V_1, \dots, V_m$). The objective is to maximize the utility for all allies while minimizing the information leakage to all adversaries. A critical innovation in EIGAN is the use of Cross-Entropy (CE) loss, which provides a more stable training signal than KL-divergence and effectively pushes adversary predictions toward a uniform distribution, representing total uncertainty.
To address data decentralization, D-EIGAN allows each node to train a local encoder. These nodes periodically synchronize their parameters with a central server. To optimize communication and enhance privacy, D-EIGAN employs sparsification, sharing only a random fraction of model weights. This technique not only reduces bandwidth but also helps mitigate local data biases (the Sattler effect), leading to a more robust global representation.
Evaluation and Results
The performance of EIGAN was evaluated across diverse benchmarks, including MNIST, MIMIC-III, and FaceScrub. On the FaceScrub dataset, EIGAN achieved a 47% improvement in ally prediction accuracy compared to the Bertran-PRL baseline, without increasing sensitive information leakage.
As illustrated in the experimental results, EIGAN (red line) consistently outperforms baselines such as standard Autoencoders, PCA, and Differential Privacy (DP) methods. It simultaneously achieves the lowest ally loss and the highest adversary loss relative to non-private baselines. Additionally, D-EIGAN demonstrated consistent performance parity with the centralized EIGAN, proving that distributed learning does not necessitate a trade-off in representation quality.
Conclusion
EIGAN and D-EIGAN represent a significant advancement in Private Representation Learning by providing a scalable, multi-attribute, and distributed framework. By leveraging adversarial training and sparsified parameter synchronization, these models offer a practical path toward privacy-preserving data sharing in complex, decentralized environments. Future research will explore the integration of formal differential privacy guarantees and the optimization of training for high-dimensional attribute sets.
Summary updated