Locating disparities in machine learning

Moritz von Zahn, Oliver Hinz, Stefan Feuerriegel

Abstract

Machine learning was repeatedly proven to provide predictions with disparate outcomes, in which subgroups of the population (e.g., defined by age, gender, or other sensitive attributes) are systematically disadvantaged. Previous literature has focused on detecting such disparities through statistical procedures for when the sensitive attribute is specified a priori. However, this limits applicability in real-world settings where datasets are high dimensional and, on top of that, sensitive attributes may be unknown. As a remedy, we propose a data-driven framework called Automatic Location of Disparities (ALD) which aims at locating disparities in machine learning. ALD meets several demands from machine learning practice: ALD (1) is applicable to arbitrary machine learning classifiers; (2) operates on different definitions of disparities (e.g., statistical parity or equalized odds); (3) deals with both categorical and continuous predictors; (4) is suitable to handle high-dimensional settings; and (5) even identifies disparities due to intersectionality where disparities arise from complex and multi-way interactions (e.g., age above 60 and female). ALD produces interpretable fairness reports as output. We demonstrate the effectiveness of ALD based on both synthetic and real-world datasets. As a result, ALD helps practitioners and researchers of algorithmic fairness to detect disparities in machine learning algorithms, so that disparate -- or even unfair -- outcomes can be mitigated. Moreover, ALD supports practitioners in conducting algorithmic audits and protecting individuals from discrimination.

Introduction

As machine learning (ML) systems are increasingly deployed in high-stakes domains such as hiring, lending, and criminal justice, the risk of systematic disadvantage to specific population subgroups—known as disparate outcomes—becomes a critical concern. Traditional fairness audits often suffer from a "needle in a haystack" problem: they require auditors to pre-specify sensitive attributes (e.g., "gender" or "race") before the audit begins.

This approach fails to capture complex, intersectional biases (e.g., bias against Black women specifically) or to identify precise thresholds in continuous variables like age. The Automatic Location of Disparities (ALD) framework addresses these gaps by providing a data-driven method to discover hidden biases without prior knowledge of which attributes are problematic. By automating the discovery of these subgroups, ALD enables more comprehensive algorithmic audits and helps ensure compliance with emerging regulations like the EU AI Act.

Methods

ALD employs a robust three-step pipeline to identify and validate disparity-inducing attributes:

  1. Recursive Partitioning (Discovery): The framework splits the dataset. On the first half, it utilizes a forest of conditional inference trees to search for subgroups. These trees identify the strongest statistical associations between predictors and model outcomes, allowing for the exploration of complex multi-way interactions.
  2. Statistical Hypothesis Testing (Validation): To prevent findings from being mere statistical artifacts, ALD validates discovered subgroups on the second half of the data. It uses $\chi^2$ tests to measure disparity magnitude (e.g., statistical parity) and applies Benjamini-Hochberg corrections to manage the multiple testing problem.
  3. Audit Report Generation: The final output is an interpretable report where subgroups are ranked by bias magnitude or statistical significance, allowing auditors to prioritize mitigation efforts.

Results and Discussion

Performance on Synthetic Benchmarks

In controlled experiments using synthetic datasets, ALD demonstrated superior performance compared to existing baselines like Multivariate Subset Scan. Specifically, ALD excelled in locating exact "cut-off" points for continuous variables and identifying "hidden" intersectional biases where disparities only emerge through the combination of multiple traits.

Performance comparison of ALD against baselines on synthetic datasets, showing success rates in locating subgroups across varying disparity levels and age spans.

Real-World Application

When applied to the Adult Income dataset (predicting earnings >$50k), ALD identified that gender bias was most pronounced for "married men" (husbands), who were significantly more likely to receive high-earning predictions. On the COMPAS recidivism dataset, it flagged not only racial disparities but also specific error rate spikes in smaller subgroups that might otherwise be overlooked.

Ranked subgroups identified by ALD in the Adult Income dataset, highlighting the intersectional nature of the discovered disparities.

Conclusion

ALD represents a significant advancement in algorithmic fairness by moving beyond a priori assumptions. While powerful, the framework is limited to attributes present in the training data and identifies associations rather than causal relationships. Future work will focus on handling missing sensitive attributes and integrating causal inference to better understand the origins of the detected biases.

Summary updated

Abstract

Machine learning was repeatedly proven to provide predictions with disparate outcomes, in which subgroups of the population (e.g., defined by age, gender, or other sensitive attributes) are systematically disadvantaged. Previous literature has focused on detecting such disparities through statistical procedures for when the sensitive attribute is specified a priori. However, this limits applicability in real-world settings where datasets are high dimensional and, on top of that, sensitive attributes may be unknown. As a remedy, we propose a data-driven framework called Automatic Location of Disparities (ALD) which aims at locating disparities in machine learning. ALD meets several demands from machine learning practice: ALD (1) is applicable to arbitrary machine learning classifiers; (2) operates on different definitions of disparities (e.g., statistical parity or equalized odds); (3) deals with both categorical and continuous predictors; (4) is suitable to handle high-dimensional settings; and (5) even identifies disparities due to intersectionality where disparities arise from complex and multi-way interactions (e.g., age above 60 and female). ALD produces interpretable fairness reports as output. We demonstrate the effectiveness of ALD based on both synthetic and real-world datasets. As a result, ALD helps practitioners and researchers of algorithmic fairness to detect disparities in machine learning algorithms, so that disparate -- or even unfair -- outcomes can be mitigated. Moreover, ALD supports practitioners in conducting algorithmic audits and protecting individuals from discrimination.
BibTeX
@inproceedings{zahn2023locating,
  author = {Moritz von Zahn and Oliver Hinz and Stefan Feuerriegel},
  title = {{Locating disparities in machine learning}},
  booktitle = {IEEE Big Data},
  year = {2023},
  doi = {10.1109/bigdata59044.2023.10386485},
}
Figure