Abstract
Adversarial representation learning is a promising paradigm for obtaining data representations that are invariant to certain sensitive attributes while retaining the information necessary for predicting target attributes. Existing approaches solve this problem through iterative adversarial minimax optimization and lack theoretical guarantees. In this paper, we first study the "linear" form of this problem i.e., the setting where all the players are linear functions. We show that the resulting optimization problem is both non-convex and non-differentiable. We obtain an exact closed-form expression for its global optima through spectral learning. We then extend this solution and analysis to non-linear functions through kernel representation. Numerical experiments on UCI and CIFAR-100 datasets indicate that, (a) practically, our solution is ideal for "imparting" provable invariance to any biased pre-trained data representation, and (b) empirically, the trade-off between utility and invariance provided by our solution is comparable to iterative minimax optimization of existing deep neural network based approaches. Code is available at https://github.com/human-analysis/kernel-adversarial-representation-learning.git
Introduction
Adversarial Representation Learning (ARL) is a critical technique for ensuring fairness in machine learning models. The goal is to learn an encoder that produces a representation $z$ which preserves information for a target task $y$ while eliminating information about a sensitive attribute $s$. Traditionally, this is framed as a minimax game between an encoder and an adversary. However, in the context of Deep Neural Networks (DNNs), this optimization is notoriously unstable, often failing to converge or hitting local optima. Furthermore, these "black box" models lack theoretical guarantees regarding the optimality of the achieved fairness-utility trade-off.
Methodology: Spectral Adversarial Representation Learning (SARL)
The authors investigate the fundamental properties of ARL by analyzing its linear form. They demonstrate that even when all components are linear, the optimization problem is non-convex and non-differentiable, explaining the instability of standard gradient-based methods.
To overcome these limitations, the authors introduce Spectral-ARL (SARL). By reformulating the objective as a trace optimization problem on a Stiefel manifold, they derive a closed-form solution that identifies the global optimum via eigendecomposition. This "one-shot" approach provides a stable and theoretically grounded alternative to iterative training. The method is further extended to non-linear data through Kernel-SARL, which utilizes the kernel trick to capture complex relationships while maintaining the stability of a closed-form solution.
One key advantage of SARL is its ability to analytically determine the optimal dimensionality for the embedding space, rather than relying on heuristic tuning.
Experimental Results
The proposed methods were evaluated on the UCI Adult, UCI German Credit, and CIFAR-100 datasets.
Performance and Invariance
On tabular datasets, SARL consistently outperformed DNN-based baselines such as VFAE and MaxEnt-ARL in the trade-off between accuracy and fairness. For instance, on the Adult dataset, SARL achieved a target accuracy of 84.1% with 0.0% leakage of the sensitive gender attribute (measured as the deviation from a random guess). This level of provable invariance is difficult to achieve with iterative minimax optimization.
Application to Pre-trained Models
The authors also demonstrated how SARL can be used to "impart" fairness to existing biased models. By applying the SARL encoder to features extracted from a pre-trained ResNet-110 on CIFAR-100, they achieved significant invariance without the need for end-to-end retraining. Kernel-SARL proved essential in this high-dimensional setting, significantly outperforming the linear version.
Conclusion
Spectral-ARL offers a robust, theoretically sound framework for learning fair representations. By providing a closed-form solution, it avoids the pitfalls of adversarial minimax games and allows for the "fixing" of biased pre-trained models. While scaling kernel methods to massive datasets remains a challenge, SARL establishes a new baseline for what is achievable in fair representation learning.
Summary updated