Does mitigating ML's impact disparity require treatment disparity?

Zachary C. Lipton, Julian McAuley, Alexandra ChouldechovaNeurIPS 2018 · OpenReview

Introduction

In the pursuit of algorithmic fairness, a fundamental tension exists between two primary goals: Treatment Parity (ensuring a model is "blind" and does not use protected attributes to make decisions) and Impact Parity (ensuring the model produces similar outcomes, such as hiring rates, across different demographic groups).

To reconcile these competing objectives, many researchers have proposed Disparate Learning Processes (DLPs). These algorithms use sensitive information during training to "de-bias" the model but produce a final classifier that remains blind to the sensitive attribute at prediction time.

This paper challenges the popular notion that machine learning models must be "blind" to protected characteristics at test time to be fair. Through theoretical proofs and empirical experiments, the authors demonstrate that DLPs are mathematically suboptimal and can inadvertently harm the very groups they are intended to protect.

Methodology and Datasets

The authors combine decision theory with empirical experiments to dissect the mechanics of DLPs. They mathematically prove that any DLP will result in lower or equal accuracy compared to a model that explicitly uses protected attributes to set group-specific thresholds.

To validate these theoretical findings, the authors compared a state-of-the-art DLP against a simple "optimal thresholding" strategy across several datasets:

  • Synthetic Hiring Data: A toy dataset where "work experience" predicts success, but "hair length" acts as a proxy for gender.
  • CS Graduate Admissions: Real data from a large US university (9,000 records) with injected synthetic bias.
  • Public Benchmarks: UCI Income (Adult), UCI Marketing, UCI Credit, and IBM Employee/Customer Attrition datasets.

Key Insights: The Hidden Costs of "Blindness"

The paper reveals several critical flaws in the DLP approach:

1. Blindness is a Proxy for Discrimination If non-sensitive features (like zip code or hobbies) are correlated with protected attributes, a "blind" DLP will simply learn to use those proxies. This effectively implements "treatment disparity" in disguise, making the model's biases less transparent rather than eliminating them.

2. The "Rational Ordering" Violation Because DLPs are forced to be blind at test time, they cannot "see" group-specific distributions. This forces the model to distort its internal logic, flipping decisions for individuals near the boundary based on irrelevant features.

Synthetic hiring dataset illustrating how a Disparate Learning Process (DLP) shifts the decision boundary, penalizing candidates based on proxy features like hair length.

As seen in the synthetic hiring simulation above, a DLP might reject a highly qualified woman because she has "male-like" short hair, while hiring a less qualified man because he has "female-like" long hair. This violates the principle of rational ordering.

3. Mathematical Suboptimality By restricting the model from using group-specific distributions at test time, DLPs yield a worse trade-off between accuracy and fairness than if the model simply used a different cutoff threshold for each group.

Empirical Results and Real-World Harm

In almost every empirical test, the optimal thresholding strategy (explicit treatment disparity) achieved higher parity for a given level of accuracy than the DLP. For instance, on the UCI Credit dataset, the DLP method dropped accuracy to 0.74 to reach an 85% parity rule. In contrast, the authors' thresholding strategy reached 100% parity while maintaining the same accuracy.

Furthermore, DLPs can violate the "do no harm" principle. Explicit treatment disparity can be constrained to only benefit the disadvantaged group. DLPs, however, often "help" the disadvantaged group by accidentally hurting its most qualified members.

Impact of DLP on graduate admissions, showing how the treatment inadvertently rejects highly qualified female candidates with "male-like" feature profiles.

In the graduate admissions case study, the DLP flipped 43 decisions to "non-admit." Alarmingly, five of these were women who were highly qualified but possessed "male-like" feature profiles, demonstrating the qualitative harm of forcing algorithmic blindness.

Limitations and Future Directions

While the paper argues for the technical superiority of explicit treatment disparity (group-specific thresholds), the authors acknowledge the legal ambiguity of this approach. In many jurisdictions, explicit thresholds may be viewed as "affirmative action," which is heavily restricted.

Additionally, the analysis assumes the underlying labels are a "ground truth" of qualification. If the training data itself is tainted by human prejudice, simply adjusting thresholds may not be sufficient.

Ultimately, the authors suggest that separating estimation (calculating an accurate probability) from decision-making (choosing the threshold) is a more robust, transparent path forward than trying to bake fairness constraints directly into a model's weights.

Summary updated

Abstract

Following precedent in employment discrimination law, two notions of disparity are widely-discussed in papers on fairness and ML. Algorithms exhibit treatment disparity if they formally treat members of protected subgroups differently; algorithms exhibit impact disparity when outcomes differ across subgroups (even unintentionally). Naturally, we can achieve impact parity through purposeful treatment disparity. One line of papers aims to reconcile the two parities proposing disparate learning processes (DLPs). Here, the sensitive feature is used during training but a group-blind classifier is produced. In this paper, we show that: (i) when sensitive and (nominally) nonsensitive features are correlated, DLPs will indirectly implement treatment disparity, undermining the policy desiderata they are designed to address; (ii) when group membership is partly revealed by other features, DLPs induce within-class discrimination; and (iii) in general, DLPs provide suboptimal trade-offs between accuracy and impact parity. Experimental results on several real-world datasets highlight the practical consequences of applying DLPs.
BibTeX
@inproceedings{lipton2018does,
  author = {Zachary C. Lipton and Julian McAuley and Alexandra Chouldechova},
  title = {{Does mitigating ML's impact disparity require treatment disparity?}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2018},
  url = {https://openreview.net/forum?id=BylUPvWuZB},
}
Figure