Introduction
Privacy-Preserving Representation Learning (PRL) has emerged as a critical field for balancing the utility of data sharing with the necessity of protecting sensitive information. This paper introduces EIGAN (Exclusion-Inclusion Generative Adversarial Network) and its distributed counterpart, D-EIGAN, representing the first frameworks capable of handling multiple sensitive attributes across siloed, distributed datasets. By reformulating PRL as a minimax game optimized via cross-entropy loss, the authors demonstrate significant improvements in both utility and privacy robustness.
The Challenge of Real-World Privacy
Modern data ecosystems face two primary hurdles in privacy preservation. First is the Complexity Gap: while most existing PRL methods focus on a single ally task and one sensitive attribute, real-world scenarios like healthcare often involve multiple concurrent objectives and various sensitive features (e.g., race, gender, age). Second is the Distribution Gap: data is frequently siloed across institutions that cannot share raw information due to regulatory constraints. Existing centralized PRL techniques are often inapplicable in these federated environments.
Methodology: EIGAN and D-EIGAN
The proposed EIGAN architecture utilizes a minimax optimization framework. An Encoder generates a data representation that a set of Ally networks uses to predict intended targets, while a set of Adversary networks attempts to recover sensitive attributes from the same representation.
Unlike previous approaches relying on KL-divergence, EIGAN employs Cross-Entropy (CE) loss. The authors argue that CE provides superior stability and more effectively pushes adversaries toward a uniform "random guess" distribution, particularly when data distributions are disjoint.
To address siloed data, D-EIGAN extends this framework to a federated learning setting. Local nodes train individual encoders and periodically synchronize parameters with a central server. To optimize communication, the framework employs sparsification—sharing only a fraction $\phi$ of parameters—and adjustable synchronization frequencies ($\delta$).
Key Results and Insights
The framework was rigorously evaluated across diverse datasets, including MNIST, MIMIC-III, Adult, and FaceScrub.
Utility and Privacy Performance
EIGAN achieves substantial performance gains. On the FaceScrub dataset, it improved ally accuracy by 47% compared to the previous state-of-the-art (Bertran-PRL) while maintaining equivalent privacy levels. On the Adult census dataset, EIGAN matched the performance of optimal closed-form linear solutions while offering the flexibility of non-linear neural architectures.
The Sparsification Paradox
A notable finding in the distributed setting is the "Sparsification Paradox." In non-i.i.d. scenarios where data distributions vary significantly between nodes, sharing fewer parameters (high sparsification) can actually improve global model performance by reducing the propagation of local data biases.
Architecture Agnosticism
The study also confirms that representations trained against one type of adversary (e.g., ResNet) remain private even when attacked by different architectures (e.g., Vision Transformers). This suggests that the privacy guarantees are fundamentally embedded within the learned representation itself.
Conclusion
EIGAN and D-EIGAN provide a robust, scalable solution for generalized privacy-preserving representation learning. By successfully navigating the trade-offs between utility and privacy in both centralized and distributed environments, this work sets a new benchmark for practical PRL applications. Future research will explore more complex non-i.i.d. distributions and the behavior of these representations in continual learning settings.
Summary updated