TL;DR
Machine learning researchers often believe they are pioneering the mathematical study of fairness, but psychometricians were there 50 years ago. This paper traces the history of "test fairness" from the 1960s to the 1970s, revealing that many modern ML fairness metrics—like Equality of Opportunity and Predictive Parity—are actually decades-old ideas rediscovered under new names.
What Problem Does This Paper Address?
As AI systems increasingly make high-stakes decisions in hiring, lending, and policing, the ML community has scrambled to define "fairness" mathematically. However, this work often happens in a vacuum. The authors argue that we are "reinventing the wheel" and, more dangerously, ignoring the lessons learned when the educational and employment testing communities faced these exact same challenges following the U.S. Civil Rights Act of 1964.
The paper addresses the gap between modern algorithmic fairness and historical psychometrics. It seeks to answer: Which modern ML metrics have historical precedents? Why did the intense interest in quantitative fairness "die out" in the late 1970s? And what can ML practitioners learn from that decline to avoid a similar fate?
Methods
The authors perform a retrospective literature review and mathematical mapping. They categorize historical fairness definitions into four buckets:
- Individual: Defined purely in terms of individuals.
- Non-comparative: Fairness for a subgroup without referencing others.
- Subgroup Parity: Parity of a specific value across groups.
- Correlation: The correlation between demographic variables and model output.
The core of their method is a formal translation (summarized in Table 3 of the paper) that maps historical criteria (like Cleary’s 1968 criterion or Cole’s 1973 definition) onto the modern ML framework of Independence, Separation, and Sufficiency.
Datasets
As a survey and theoretical paper, it does not introduce new datasets or benchmarks. It references historical data studies, such as T. Anne Cleary’s 1960s research comparing SAT scores and college GPAs across racial groups, and the controversial "race-norming" practices in U.S. employment services in the 1980s.
Key Insights and Ideas
- "History Rhymes": Many "novel" ML breakthroughs are echoes. For example, the "Equality of Opportunity" metric popularized by Hardt et al. in 2016 is mathematically identical to Nancy Cole’s 1973 "conditional probability" definition.
- The Property Conflict: A major historical debate was whether fairness is a property of the test itself (the model) or the use of the test (the decision threshold). Cleary argued for the former, while Thorndike argued for the latter. ML still struggles with this distinction today.
- Differential Item Functioning (DIF): Psychometricians had a tool called DIF to find bias in individual test questions (features), not just the final score. The authors note that ML currently lacks a robust analog for "unfairness interpretability" at the feature level.
- The Impossibility Results: The 1970s researchers already knew that you cannot satisfy all fairness definitions simultaneously unless groups have identical base rates. This "impossibility" is often treated as a recent discovery in ML (e.g., the COMPAS debate).
- The Value of Values: The 1970s "Fairness Tide" turned because researchers realized that math cannot solve a political problem. Technical definitions encode specific value systems (e.g., utilitarianism vs. individual rights), and without explicit ethical framing, the math becomes a distraction.
Key Results
The paper provides a definitive "Rosetta Stone" for fairness research. Key mappings include:
- Cole (1973) $\approx$ Equality of Opportunity (Equal True Positive Rates).
- Linn (1973) $\approx$ Predictive Parity (Equal Positive Predictive Value).
- Peterson & Novick (1976) $\approx$ Equalized Odds (Equal TPR and TNR).
- Darlington (1971) $\approx$ Demographic Parity (in specific Gaussian cases).
Qualitatively, the authors show that the decline of fairness research in the late 70s was due to a lack of consensus and a divergence between technical definitions and public/legal perceptions of justice.
Limitations and Future Work
The authors acknowledge that their review is primarily focused on the U.S. legal and cultural context (specifically the fallout of the Civil Rights Act).
For future work, they suggest the ML community should:
- Move beyond "What is a fair model?" to "What causes the disparities?" (Causal pathways).
- Develop ML-DIF: Create methods to audit individual model features for bias, rather than just the final output.
- Incorporate Context: Stop treating models as static objects and start measuring fairness in the context of their real-world deployment and decision thresholds.
- Explicit Ethics: Stop pretending fairness metrics are value-neutral. Researchers must explicitly state which ethical theories (like proportional representation or individualism) their chosen metrics support.
Summary updated