Algorithmic Unfairness Through the Lens of EU Non-Discrimination Law

Hilde J. P. Weerts, Raphaële Xenidis, Fabien Tarissan, Henrik Palmer Olsen, Mykola PechenizkiyEWAF 2023 · OpenReview

Introduction

As algorithmic systems increasingly mediate access to essential services—from childcare benefits to employment—the technical community has turned toward "algorithmic fairness" as a solution to bias. However, this research often hits a wall when confronted with the realities of European Union (EU) non-discrimination law. This paper, originally presented as an extended abstract at the 2023 ACM Conference on Fairness, Accountability, and Transparency (FAccT), explores the friction between technical optimization and legal interpretation. It argues that bridging this gap requires moving beyond mathematical metrics to address the normative justifications behind the distribution of benefits and burdens.

The Disconnect: Metrics vs. Context

A significant "lost in translation" effect persists between computer science (CS) and legal scholarship. In the CS perspective, fairness is often treated as a set of discrete mathematical constraints, such as demographic parity or equalized odds. Researchers frequently seek a "gold standard" metric that can guarantee legal compliance through technical intervention.

In contrast, the legal perspective views non-discrimination law as "polysemous"—its concepts carry multiple meanings that shift depending on the specific context and the protected interest at stake. Unlike a computer program, EU law does not function as a "decision tree" or a flowchart with fixed thresholds. Instead, courts interpret equality on a case-by-case basis, making a direct translation into code impossible. This leads to a mutual overestimation: lawyers may overestimate the power of technical "debiasing," while computer scientists may place undue faith in the "legal certainty" and determinacy of the law.

Methodology and Case Analysis

The authors utilize an interdisciplinary framework to map the requirements of EU law against algorithmic fairness research. Their approach combines legal analysis with computer science theory through several lenses:

  1. Taxonomy of Harms: Categorizing discriminatory harms through high-profile failures.
  2. Comparative Mapping: Aligning legal requirements with existing algorithmic fairness interventions.
  3. Case Law Analysis: Examining landmark decisions from the Court of Justice of the European Union (CJEU) to identify the "normative underpinnings"—the hidden values and assumptions—that guide legal reasoning.

The study grounds its theoretical analysis in three real-world examples of algorithmic failure:

  • The Dutch Childcare Benefits Scandal: A case involving algorithmic profiling for fraud detection that resulted in systemic discrimination.
  • Amazon’s Recruitment Tool: An experimental AI that penalized resumes containing the word "women's," illustrating how models mirror historical human bias.
  • Gender Shades: The seminal benchmark study by Buolamwini and Gebru demonstrating intersectional accuracy disparities in commercial facial recognition technology.

Key Insights: The "Emptiness" of Equality

A central theme of the paper is the "emptiness of equality." In legal theory, equality is considered an empty concept until a baseline is established. To determine if two groups are treated "equally," one must first decide which characteristics are relevant for the comparison. These decisions are normative and value-based, not mathematical. Both CS metrics and legal arguments rely on these implicit assumptions, and the authors argue they must be made explicit to facilitate translation between the fields.

Furthermore, the authors highlight the "prima facie" hurdle. Under EU law, a claimant must provide initial evidence of a disadvantage to proceed with a discrimination claim. However, the inherent opacity of AI systems makes it exceptionally difficult for victims to meet this burden of proof, as they often cannot see or understand the logic that led to a negative outcome.

Conclusion: From "What" to "Why"

The research concludes that while EU law is well-equipped to handle algorithmic unfairness that mirrors traditional human discrimination (as seen in the Amazon case), it struggles with "disparate predictive performance"—cases where a model is simply less accurate for one group than another.

Ultimately, the paper calls for a shift in focus. AI practitioners should stop asking "What metric should we equalize?" and instead ask "Why is this specific distribution of outcomes justified in this context?" Because EU law is fundamentally contextual, non-discrimination compliance cannot be fully automated. Future efforts should focus on creating frameworks that help practitioners navigate normative goals and provide the transparency necessary for legal accountability in an era of opaque systems.

Summary updated

BibTeX
@inproceedings{weerts2023algorithmic,
  author = {Hilde J. P. Weerts and Raphaële Xenidis and Fabien Tarissan and Henrik Palmer Olsen and Mykola Pechenizkiy},
  title = {{Algorithmic Unfairness Through the Lens of EU Non-Discrimination Law}},
  booktitle = {EWAF},
  year = {2023},
  url = {https://openreview.net/forum?id=i8Kwho8AAo},
}
Figure