Uncertainty Matters: Stable Conclusions under Unstable Assessment of Fairness Results

Ainhize Barrainkua, Paula Gordaliza, José Antonio Lozano, Novi QuadriantoAISTATS 2024

Abstract

Fairness metrics in machine learning are notoriously unstable, often swinging wildly based on how a dataset is split. This paper introduces Uncertainty Matters (UM), a Bayesian framework that treats performance and fairness metrics as random variables to provide reliable, probabilistic comparisons between algorithms, even when traditional cross-validation yields contradictory results.

Introduction: The Problem of Brittle Fairness Metrics

In the quest for "fair" AI, researchers compare algorithms using metrics like Accuracy and Equality of Opportunity (EOp). The standard practice is to use $K$-fold cross-validation (CV) and report the average. However, fairness metrics are highly sensitive to data partitions. Because fairness is often calculated on small minority subgroups, a single "unlucky" data split can make a mediocre algorithm look like a superstar, or vice versa.

The high variance of fairness and accuracy metrics across different model types illustrates the instability of point estimates.

If you run a 10-fold CV twice with different random seeds, you might reach two different conclusions about which model is "better." This instability leads to a lack of reproducibility and "brittle" conclusions in fairness research.

The Uncertainty Matters (UM) Framework

The authors propose the Uncertainty Matters (UM) framework, which moves away from deterministic point estimates toward a full probabilistic treatment of model performance.

1. Probabilistic Confusion Matrix

Instead of treating the confusion matrix (counts of True Positives, False Negatives, etc.) as fixed, they model it using a Dirichlet-Multinomial distribution. This allows them to treat any metric derived from the confusion matrix (like accuracy or EOp) as a random variable with its own posterior distribution.

2. Handling Correlated Folds

In $K$-fold CV, the folds are not independent because they share training data. The authors introduce an "Effective Confusion Matrix" that mathematically accounts for this correlation ($\rho$).

3. Complexity-Aware Correlation

They recognize that different models have different stability levels. They propose a novel method to estimate the correlation parameter based on the model's complexity relative to a reference model (like an SVM).

4. Region of Practical Equivalence (RoPE)

To compare two models, they don't just ask "is A > B?" Instead, they define a small window (RoPE) where differences are considered negligible. They then calculate the probability that A is better than B, B is better than A, or they are practically equivalent across multiple objectives simultaneously.

Experimental Evaluation

The framework was evaluated primarily on the German Credit Dataset (1,000 instances). Its moderate size naturally introduces uncertainty, allowing the authors to run 10,000 different CV configurations to establish a "ground truth" probability distribution. The Adult Income Dataset was also used for case studies to demonstrate how UM can refine conclusions drawn from standard evaluations.

Key Results

The UM framework's estimated probabilities for events (e.g., "Algorithm A outperforms B") closely align with the empirical ground truth derived from 10,000 repeated experiments.

Comparison of Ground Truth distributions with standard 10-fold CV and the proposed UM framework. UM (c, f) provides a much more accurate estimation of the true distribution (a, d) than standard CV (b, e).

Key findings include:

  • Superiority over SOTA: Compared to existing state-of-the-art Bayesian evaluation methods, UM provides more accurate uncertainty estimates, especially for complex models where standard methods tend to be overconfident.
  • Flipping Conclusions: In case studies on the Adult dataset, the authors showed that a deterministic evaluation might say "Algorithm A is better," while the UM framework reveals there is actually a 77% probability that "Algorithm B" is the true winner when uncertainty is accounted for.
  • Multi-Objective Nuance: By using RoPE, the framework acknowledges that a 0.001 difference in accuracy isn't a "win"—it's a tie. This leads to more honest reporting of algorithmic superiority.

Discussion and Future Work

While powerful, UM has some limitations. Estimating the correlation parameter $\rho$ requires additional half-sized CV runs, which could become computationally expensive for massive datasets. Additionally, as the number of metrics increases, most algorithms become "Pareto-optimal," making clear dominance harder to establish. Future work may explore integrating unlabeled data to further reduce uncertainty and extending the framework to handle noisy or missing sensitive attribute information.

Summary updated

BibTeX
@inproceedings{barrainkua2024uncertainty,
  author = {Ainhize Barrainkua and Paula Gordaliza and José Antonio Lozano and Novi Quadrianto},
  title = {{Uncertainty Matters: Stable Conclusions under Unstable Assessment of Fairness Results}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2024},
}
Figure