Variance, Self-Consistency, and Arbitrariness in Fair Classification

A. Feder Cooper, Solon Barocas, Christopher De Sa, Siddhartha Sen

Introduction: The Hidden Arbitrariness in Algorithmic Decisions

In algorithmic fairness research, the standard practice is to train a single model and evaluate its performance across demographic groups. However, this approach overlooks model multiplicity: the phenomenon where numerous distinct models can achieve similar overall accuracy while making conflicting predictions for the same individual.

When a learning process produces models that disagree on a specific person's outcome based on minor variations in training data, that person's decision is effectively arbitrary. This paper argues that such statistical variance is a hidden confounder in fairness research. If a decision is essentially a coin flip due to model instability, the process lacks reliability, regardless of whether it satisfies traditional parity metrics.

Figure 1: Model multiplicity leads to arbitrary outcomes for certain individuals. While Individual 1 receives a consistent prediction across models, Individual 2's outcome is effectively a coin flip.

Methodology: Quantifying and Mitigating Variance

To address this, the authors shift focus from individual models to the distribution of possible models. They introduce two core concepts:

  1. Self-Consistency (SC): A metric derived from statistical variance that measures the probability that two models from the same learning process will agree on a prediction for a specific instance. Low SC serves as a proxy for high arbitrariness.
  2. SC Ensembling with Abstention: A modified bagging approach where the system calculates the SC for an individual. If the SC falls below a user-defined threshold ($\kappa$), the model abstains from making a prediction, deferring the case to human experts or non-automated processes.

The authors propose two ensembling variants: Simple Ensembling, which uses standard bagging with an abstention threshold, and Super Ensembling, a two-stage process that uses bagging to reduce variance before ensembling those bagged models to decide on abstention. Super ensembling significantly reduces the number of cases where the model must abstain while maintaining high consistency.

Key Insights: Variance as a Confounder

The study's most significant finding is that many "unfairness" gaps reported in literature may actually be artifacts of high statistical variance rather than stable properties of the datasets. By using ensembling to "average away" the noise of individual models, the authors discovered that many standard benchmarks are actually close-to-fair before any explicit fairness interventions are applied.

This suggests a potential reliability crisis in fairness research: high variance, particularly in smaller datasets like COMPAS or German Credit, can lead researchers to misidentify unstable model behavior as systemic bias.

Empirical Results

The proposed ensembling methods achieved state-of-the-art results across several benchmarks. On the COMPAS dataset, the method reduced the False Positive Rate disparity ($\Delta FPR$) to just 1.8–3%, far lower than typical single-model baselines. Furthermore, because the model abstains on high-variance "hard" cases, the accuracy on the remaining prediction set increases dramatically. For instance, on the Old Adult dataset, error rates dropped from approximately 17% to 8% within the prediction set.

Figure 6: Comparison of fairness metrics across baseline and ensembling methods. Both Simple and Super ensembling significantly reduce False Negative Rate disparities ($\Delta FNR$) compared to baseline models.

Conclusion and Future Work

By accounting for arbitrariness, this research calls into question the necessity of complex fairness-improving algorithms for many standard tasks. While the cost of abstention is a practical limitation, the authors suggest that identifying arbitrary decisions is a prerequisite for fair and due-process-compliant automated systems. Future work will extend these concepts to deep learning and explore the legal implications of algorithmic arbitrariness.

Summary updated

BibTeX
@inproceedings{cooper2024variance,
  author = {A. Feder Cooper and Solon Barocas and Christopher De Sa and Siddhartha Sen},
  title = {{Variance, Self-Consistency, and Arbitrariness in Fair Classification}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i20.30203},
}
Figure