Invariant Representations with Stochastically Quantized Neural Networks

Mattia Cerrato, Marius Köppel, Roberto Esposito, Stefan KramerAAAI 2023 · DOI

Introduction

Algorithmic fairness has become a critical frontier in machine learning. As models are increasingly deployed in high-stakes domains like finance and criminal justice, the risk of inheriting and amplifying historical biases is significant. A common approach to mitigating this is learning invariant representations—features that retain task-relevant information while being statistically independent of sensitive attributes (e.g., race, gender).

The Challenge of Mutual Information

The core technical hurdle in learning invariant representations is minimizing the Mutual Information (MI) between the model’s internal state and the sensitive attribute $S$. In standard deterministic neural networks, MI is notoriously difficult to estimate; it is often either infinite or mathematically "vacuous." Existing methods typically rely on adversarial training—where a secondary network attempts to predict the sensitive attribute—or complex variational approximations. However, adversarial methods are often unstable, difficult to tune, and prone to "cat-and-mouse" dynamics that hinder convergence.

Methodology: Stochastic Quantization

To overcome these limitations, the authors propose a shift from continuous, deterministic activations to Stochastically Quantized Neural Networks.

Architecture of the Stochastically Quantized Neural Network showing the stochastic layer and the combined loss function.

The proposed architecture introduces a "stochastic layer" where neurons act as discrete random variables. Instead of outputting a continuous value, each neuron fires with a probability determined by a sigmoid function, effectively becoming a Bernoulli random variable. This discretization serves as a mathematical bridge: because the neurons are discrete, the MI between the layer and the sensitive attribute $S$ can be calculated analytically using closed-form formulas.

The objective function balances task accuracy (via a standard cross-entropy or KL-divergence loss) with a penalty for the MI found in the stochastic layer. The authors prove that minimizing the sum of MI for individual neurons provides a valid upper bound for the total MI of the layer, simplifying a complex high-dimensional optimization into a series of tractable 1D calculations.

Experimental Evaluation

The method was evaluated across several benchmarks:

  • COMPAS & Adult: Real-world datasets for recidivism and income prediction, focusing on fairness regarding ethnicity and gender.
  • Bank Marketing: Predicting subscriptions while maintaining age-invariance.
  • Biased-MNIST: A synthetic test where models must ignore background color to identify digits, testing the ability to discard "nuisance" information.

The study compared two variants: BinaryBernoulli (using analytical formulas) and BinaryMI (using density estimation).

Key Results

The stochastically quantized approach demonstrated superior performance and stability compared to traditional adversarial and variational models.

Accuracy vs. Fairness trade-off on the COMPAS dataset comparing BinaryMI and BinaryBernoulli to baselines.

On the COMPAS dataset, BinaryMI achieved a better trade-off between accuracy (AUC) and fairness (measured by Group Probability Accuracy, GPA) than state-of-the-art baselines like VFAE and AdvCls. Furthermore, the relationship between the fairness trade-off parameter $\gamma$ and the resulting model fairness was found to be highly linear and predictable (correlation $\rho = 0.924$), a significant advantage for practitioners who must adhere to specific regulatory thresholds.

In the Biased-MNIST task, the BinaryBernoulli model achieved 96.02% accuracy, significantly outperforming the vanilla model (89.1%) by successfully ignoring the misleading background color correlation present in the training data.

Conclusion

By leveraging stochastic quantization, this work transforms Mutual Information from a theoretical obstacle into a practical tool for fairness. The transition to discrete, stochastic activations allows for exact analytical calculations, replacing unstable adversarial games with a smooth, predictable optimization process. This approach not only improves fairness/accuracy trade-offs but also provides a robust framework for learning representations invariant to any specified nuisance factor.

Summary updated

BibTeX
@inproceedings{cerrato2023invariant,
  author = {Mattia Cerrato and Marius Köppel and Roberto Esposito and Stefan Kramer},
  title = {{Invariant Representations with Stochastically Quantized Neural Networks}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i6.25851},
}
Figure