Locating disparities in machine learning

Moritz von Zahn, Oliver Hinz, Stefan Feuerriegel

TL;DR

Researchers have developed Automatic Location of Disparities (ALD), a framework that automatically identifies subgroups of a population—such as "women over 60"—that are systematically disadvantaged by machine learning models. Unlike previous tools, ALD doesn't need to be told which attributes to check; it discovers complex, "hidden" biases (intersectionality) and handles continuous data like age with high precision.

What Problem Does This Paper Address?

As machine learning (ML) moves into high-stakes areas like hiring, lending, and criminal justice, the risk of "disparate outcomes" grows. While we have metrics to measure fairness, most existing tools suffer from a "needle in a haystack" problem: they require practitioners to specify exactly which groups to check (e.g., "check for gender bias") before the audit begins.

In the real world, bias is rarely that simple. It often hides in intersectionality—where a model might be fair to women in general and fair to Black people in general, but systematically disadvantages Black women. Furthermore, attributes like "age" are continuous; a model might be biased against people over 55, but a human auditor might only think to check "over 65." ALD aims to automate this discovery process, making it easier for companies to comply with new regulations like the EU AI Act.

Methods

ALD uses a robust three-step pipeline to find these "disparity-inducing" attributes:

  1. Recursive Partitioning (Discovery): The framework splits the data in half. On the first half, it uses a "forest" of conditional inference trees to search for subgroups. These trees branch out by looking for the strongest statistical associations between attributes (like age, race, or gender) and the model's outcome. Because it uses a forest of randomized trees, it can explore many different combinations of traits simultaneously.
  2. Statistical Hypothesis Testing (Validation): To ensure the discovered subgroups aren't just statistical flukes, ALD tests them against the second half of the data. It uses $\chi^2$ tests to measure the magnitude of the disparity (e.g., statistical parity or equalized odds) and applies the Benjamini-Hochberg correction to account for the "multiple testing" problem.
  3. Audit Report Generation: Finally, it produces an interpretable report. Subgroups are ranked by either the magnitude of the bias or the confidence (statistical significance) of the finding, allowing auditors to prioritize their investigations.

Datasets

The authors evaluated ALD using both controlled environments and messy real-world data:

  • Synthetic Datasets: Two custom datasets were created to test specific challenges. One focused on finding the exact "cut-off" point in a continuous variable (age), and the other featured "hidden" intersectionality where bias only appeared when gender and race were combined.
  • Adult Income Dataset: A classic benchmark used to predict whether individuals earn over $50,000.
  • COMPAS Dataset: A controversial dataset used to predict criminal recidivism, frequently studied for racial bias.

Key Insights and Ideas

  • Moving Beyond "A Priori" Knowledge: The most powerful aspect of ALD is that it is data-driven. It removes the human blind spots that occur when auditors only check for the biases they expect to find.
  • Beating Simpson’s Paradox: Disparities can disappear at the population level but scream for attention at the subgroup level. ALD’s tree-based approach is specifically designed to "drill down" into these nested groups.
  • The Power of the Split-Sample: By using one set of data to find the groups and a completely different set to validate them, ALD avoids "p-hacking" or overestimating the bias, providing practitioners with valid confidence measures.

Key Results

  • Superior Accuracy: In synthetic tests, ALD significantly outperformed existing baselines like "Multivariate Subset Scan." For example, in identifying intersectional bias, ALD maintained a high success rate while baselines struggled to find the "hidden" combinations.
  • Precision with Continuous Data: ALD was the only tool that reliably located the correct "split values" for continuous attributes (like identifying exactly which age range was affected).
  • Real-World Discovery: On the Adult Income dataset, ALD didn't just find gender bias; it identified that the disparity was most pronounced for "married men" (husbands), who were 23.5% more likely to be predicted as high-earners. On COMPAS, it flagged not only African American defendants but also a small "Other" race subgroup that suffered from significantly higher error rates.

Limitations and Future Work

The authors are transparent about several constraints:

  • Observed Attributes Only: ALD can only find bias in attributes that are actually in the dataset. If a model is biased based on sexual orientation but that data wasn't collected, ALD cannot find it.
  • Correlation vs. Causation: The tool identifies associations. It tells you who is being treated unfairly, but it doesn't explain why (e.g., it doesn't distinguish between a biased algorithm and biased underlying social structures).
  • Pipeline Agnostic: While ALD finds the bias in the final output, it doesn't pinpoint where in the ML pipeline (data collection, feature engineering, or model training) the bias was introduced.

The authors suggest that future research should focus on "missing" sensitive attributes and developing methods to move from associative findings to causal explanations.

Summary updated

Abstract

Machine learning was repeatedly proven to provide predictions with disparate outcomes, in which subgroups of the population (e.g., defined by age, gender, or other sensitive attributes) are systematically disadvantaged. Previous literature has focused on detecting such disparities through statistical procedures for when the sensitive attribute is specified a priori. However, this limits applicability in real-world settings where datasets are high dimensional and, on top of that, sensitive attributes may be unknown. As a remedy, we propose a data-driven framework called Automatic Location of Disparities (ALD) which aims at locating disparities in machine learning. ALD meets several demands from machine learning practice: ALD (1) is applicable to arbitrary machine learning classifiers; (2) operates on different definitions of disparities (e.g., statistical parity or equalized odds); (3) deals with both categorical and continuous predictors; (4) is suitable to handle high-dimensional settings; and (5) even identifies disparities due to intersectionality where disparities arise from complex and multi-way interactions (e.g., age above 60 and female). ALD produces interpretable fairness reports as output. We demonstrate the effectiveness of ALD based on both synthetic and real-world datasets. As a result, ALD helps practitioners and researchers of algorithmic fairness to detect disparities in machine learning algorithms, so that disparate -- or even unfair -- outcomes can be mitigated. Moreover, ALD supports practitioners in conducting algorithmic audits and protecting individuals from discrimination.
BibTeX
@inproceedings{zahn2023locating,
  author = {Moritz von Zahn and Oliver Hinz and Stefan Feuerriegel},
  title = {{Locating disparities in machine learning}},
  booktitle = {IEEE Big Data},
  year = {2023},
  doi = {10.1109/bigdata59044.2023.10386485},
}

Loading related content...

Figure