Most machine learning research treats training labels as an objective "ground truth." We assume that if a dataset says a loan was denied or a patient was healthy, that label is a perfect reflection of reality. However, a growing body of research argues that these labels are often "observed proxies" corrupted by historical prejudice and systemic error. When the benchmark itself is biased, our standard tools for measuring and enforcing fairness can actually make models less equitable while appearing to do the opposite.
The Latent Truth and the Biased Proxy
The core challenge in this field is the gap between the latent ground truth (the actual merit or state of an individual) and the observed label (what was recorded in the data). Yunyi Li in Label Bias: A Pervasive and Invisibilized Problem formalizes this as a systematic disparity where the error distribution is conditioned on demographic group membership.
Imagine a medical AI trained to predict "health" based on healthcare spending. As researchers found using Medical Expenditure Panel Survey data, Black patients are often sicker than White patients at the same spending level due to systemic barriers to care. If the model uses "spending" as a proxy for "need," the label itself is biased. A model that perfectly predicts this label isn't "accurate" in a clinical sense; it is merely a high-fidelity reproduction of a biased system.
The Illusion of the Fairness Accuracy Trade-off
A common refrain in AI ethics is that we must sacrifice accuracy to achieve fairness. Michael L. Wick, Swetasudha Panda, and Jean-Baptiste Tristan challenge this in Unlocking Fairness: a Trade-off Revisited. They argue that the "trade-off" is often an artifact of biased evaluation data.
When we evaluate a model against biased labels, a model that corrects for that bias will appear less accurate because it deviates from the flawed "truth." However, if we could measure performance against the true, uncorrupted signal, we would find that the fairest model is often the most accurate. Sanghamitra Dutta and colleagues use information theory to show that in an ideal world, fairness and accuracy are in accord; the perceived conflict only arises because our "yardstick" (the dataset) is bent.
Why Standard Metrics Fail
Standard fairness criteria, such as "Equalized Odds" introduced by Moritz Hardt, Eric Price, and Nathan Srebro, rely on balancing error rates across groups. But these metrics assume the target labels are correct. If the labels themselves are the source of the bias, enforcing parity against them can "invisibilize" discrimination.
As Yunyi Li and Maytal Saar-Tsechansky demonstrate in When More Data Lead Us Astray, if you use biased labels to identify which groups need help, you might accidentally allocate resources to the advantaged group. This creates a recursive loop: the model aligns with the biased labels, the fairness metrics report that the model is equitable, and the underlying societal disparity is reinforced and legitimized by the "fair" algorithm.
Fairness as a Corrective Lens
If labels are corrupted, how do we train better models? One perspective is to treat fairness constraints not as a "tax," but as a corrective lens. Avrim Blum and Kevin Stangl argue in Recovering from Biased Data that fairness constraints can help a model ignore the "noise" of human prejudice to find the true underlying patterns.
Other researchers suggest leveraging unlabeled data to bypass biased human annotators. Frameworks like SSFair by Vahid Noroozi and others use semi-supervised learning to break the dependence on small, biased datasets. Similarly, Emilio Dorigatti and colleagues use uncertainty-aware pseudo-labeling to prevent models from inheriting the "confirmation bias" present in initial training sets.
Moving Toward Validity
The shift in this research direction is away from purely mathematical parity and toward technical validity. As Maanas Kumar Sharma argues in To Be, Or Not To Be?, focusing solely on making a system "fair" can sometimes legitimize "snake oil" technologies that shouldn't exist at all.
True progress requires auditing the data-generating process itself. Whether it is resolving ambiguous pronouns or diagnosing chest X-rays, the consensus is clear: we cannot achieve fairness through algorithmic constraints alone if we do not first address the integrity of the labels we use to define "truth."