TL;DR
This paper investigates how common fairness criteria in machine learning hold up when the training data itself is flawed due to labeling or measurement errors. They analytically and empirically show that some fairness criteria are surprisingly robust to certain types of data bias, while others can be significantly violated, leading to worse outcomes than not using fairness constraints at all. This work provides crucial guidance for selecting appropriate fairness metrics when dealing with imperfect real-world data.
What Problem Does This Paper Address?
Machine learning models are increasingly used in high-stakes decision-making, such as loan approvals and hiring. To prevent these models from perpetuating societal biases, researchers have developed numerous "fairness criteria" (e.g., Demographic Parity, Equalized Odds). However, these criteria are typically developed and evaluated assuming that the training data is a perfect reflection of reality. In practice, datasets are often riddled with biases stemming from errors made by human annotators (labeling errors) or inaccuracies in how features are collected (measurement errors). This paper tackles the critical question: how do existing fairness criteria perform when trained on such imperfect, biased data? Do they still guarantee fairness, or can they even exacerbate unfairness?
Methods
The authors employ a two-pronged approach:
- Analytical Framework: They develop a formal model of a firm making binary decisions (accept/reject) based on agent features and a hidden qualification state. They analyze how different types of data biases (labeling errors on qualified disadvantaged individuals, labeling errors on unqualified advantaged individuals, and feature measurement errors on disadvantaged individuals) affect the optimal decision thresholds under various fairness constraints (Demographic Parity, True Positive Rate Parity, False Positive Rate Parity, and Equalized Odds). They use mathematical proofs and lemmas to derive theoretical guarantees about the robustness of these criteria.
- Numerical Experiments: To validate their analytical findings, they conduct experiments on three real-world datasets (FICO, Adult, and German credit score) and a synthetic dataset. They simulate different types of data biases and measure the resulting fairness violations, changes in decision thresholds, and impact on the firm's utility (or accuracy, in the case of Adult and German datasets).
Datasets
The paper uses the following datasets for their experiments:
- FICO Credit Score Dataset: Used to study the impact of labeling errors and feature measurement errors.
- Adult Dataset: Used to study the impact of labeling errors, with a focus on classification accuracy.
- German Credit Dataset: Also used to study the impact of labeling errors, with a focus on classification accuracy.
- Synthetic Dataset: Used to further illustrate the impact of feature measurement errors.
The authors treat the original versions of these datasets as the "unbiased ground truth" and then introduce simulated biases to observe their effects.
Key Insights and Ideas
- Not all fairness criteria are created equal: The paper reveals significant differences in how robust various fairness criteria are to data biases. Some criteria, like Demographic Parity (DP) and True Positive Rate (TPR) parity, show surprising resilience to certain types of labeling errors on disadvantaged groups.
- Bias can break fairness guarantees: While DP and TPR might still be met under certain labeling biases, the resulting decision thresholds are often different from the unbiased case, leading to a loss of utility for the decision-maker. For other criteria like False Positive Rate (FPR) parity, the violation can be substantial.
- Fairness can sometimes improve accuracy on biased data: Counterintuitively, in some scenarios with biased training data, applying fairness constraints can lead to higher accuracy (or utility) compared to a model optimized solely for accuracy. This is attributed to a regularization effect, where fairness constraints prevent overfitting to the biased training set.
- Data debiasing efforts become more important with fairness constraints: When fairness criteria couple the decision rules between groups, biases in one group's data can impact the other. This interdependence can increase a firm's incentive to invest in data debiasing for all groups.
- Understanding the type of bias matters: The robustness of a fairness criterion depends heavily on the specific form of data bias (labeling vs. measurement, and which group is affected). This highlights the need for domain-specific knowledge when selecting fairness metrics.
Key Results
- Demographic Parity (DP) and True Positive Rate (TPR) parity are robust to underestimation of qualification profiles of the disadvantaged group. This means that even with biased labels for qualified individuals in the disadvantaged group, these criteria can still be satisfied by the resulting model.
- False Positive Rate (FPR) parity is generally not robust to labeling biases on the disadvantaged group.
- Feature measurement errors on the disadvantaged group tend to violate fairness constraints, unless the errors are very specific (e.g., only affecting one group's selection rate).
- In experiments with FICO, Adult, and German credit score datasets, the analytical findings were largely supported. For instance, DP and TPR showed better robustness to labeling errors on disadvantaged groups compared to FPR.
- The paper demonstrates scenarios where fairness-constrained models can achieve higher accuracy than unconstrained models when trained on biased data. This is shown in Figure 3 for the Adult and German datasets.
Limitations and Future Work
- Focus on specific biases: The paper primarily focuses on two types of labeling biases and one type of feature measurement bias. Other forms of data bias (e.g., feedback loops, sampling bias) are not directly addressed.
- Analytical tractability: While the analytical framework provides strong theoretical insights, it relies on certain assumptions (e.g., one-dimensional features, threshold policies) that might not hold in all real-world applications.
- Normative vs. empirical trade-offs: The paper highlights that while fairness criteria are often chosen based on normative ideals, their practical performance can be drastically affected by data quality. The authors suggest that understanding these impacts should guide criterion selection.
- Future Work: The authors suggest further investigation into the interplay of different types of data biases, exploring more complex feature spaces, and developing new fairness criteria that are inherently more robust to data imperfections. They also point to the need for more sophisticated data debiasing strategies tailored to specific fairness criteria and bias types.
Summary updated