Retiring Adult: New Datasets for Fair Machine Learning

Frances Ding, Moritz Hardt, John Miller, Ludwig Schmidt

Introduction

For decades, the "UCI Adult" dataset has served as the primary benchmark for algorithmic fairness research. Derived from 1994 US Census data, it has been used to evaluate hundreds of fairness-constrained algorithms. However, as machine learning matures, the limitations of this aging dataset have become increasingly apparent. This paper argues that UCI Adult is outdated and potentially misleading, proposing a new suite of datasets called folktables to provide a more robust foundation for modern fairness research.

The Problem with UCI Adult

The ubiquity of UCI Adult in fairness literature masks several critical flaws identified by the authors:

  1. Temporal Obsolescence: The data is nearly 30 years old. Economic and social structures have shifted significantly since 1994, making it a poor proxy for modern decision-making systems.
  2. Arbitrary Binarization: The dataset uses a fixed $50,000 income threshold to define the "positive" outcome. In 1994, this threshold represented the 76th quantile for the general population but the 89th for women, creating an inherent skew that complicates fairness evaluations.
  3. Lack of Contextual Diversity: UCI Adult provides a single snapshot, ignoring how geographic and temporal variations influence both algorithmic performance and bias.

Methods: Dataset Archaeology and Folktables

To address these issues, the authors employed a two-pronged strategy. First, they performed "dataset archaeology," meticulously reconstructing the original UCI Adult dataset from the IPUMS (Integrated Public Use Microdata Series). This reconstruction, termed IPUMS Adult, allowed them to test the sensitivity of fairness metrics to the choice of income threshold.

Second, they developed the folktables Python package. This tool interfaces with the American Community Survey (ACS) Public Use Microdata Sample (PUMS), enabling researchers to generate large-scale datasets for various prediction tasks across all 50 US states and multiple years (2014–2018).

New Benchmarks

The folktables framework introduces five new prediction tasks that reflect modern social domains. These datasets are orders of magnitude larger than the original UCI Adult (which contains ~48,000 rows) and include rich geographic and temporal metadata.

Table 1: Overview of the five new prediction tasks introduced via the folktables package, showing significantly larger scale than the original Adult dataset.

The tasks include:

  • ACSIncome: Predicts if an individual’s income exceeds $50,000.
  • ACSPublicCoverage: Predicts if a low-income individual has public health insurance.
  • ACSMobility: Predicts if a young adult moved addresses in the last year.
  • ACSEmployment: Predicts employment status.
  • ACSTravelTime: Predicts if a commute exceeds 20 minutes.

Key Insights: The Threshold Effect

One of the paper's most striking findings is the "threshold effect." By using the reconstructed IPUMS Adult data, the authors demonstrated that fairness interventions are extremely sensitive to the target definition.

Figure 1: Accuracy and fairness violations (DP and EO) as a function of the income threshold in the reconstructed IPUMS Adult dataset.

As shown in Figure 1, changing the income threshold from $50,000 to $25,000 can completely alter the observed fairness violations and even flip which intervention appears most effective. This suggests that conclusions drawn from the arbitrary $50k threshold in UCI Adult may not generalize to other economic contexts.

Geographic and Temporal Dynamics

Beyond thresholds, the authors highlight that fairness is highly context-dependent:

  • Geography as a Variable: A model optimized for fairness in one state (e.g., California) may exhibit significant disparities when applied in another (e.g., Texas). The "cost" of fairness—the accuracy trade-off required to achieve parity—varies widely across regions.
  • Stability Over Time: While predictive accuracy often degrades over years due to distribution shift, fairness criteria like demographic parity appear more stable over time than across different geographies.
  • Scale and Disparity: Larger datasets do not automatically mitigate bias. Disparities in these modern datasets often reflect persistent structural inequalities rather than simple sampling issues.

Key Results and Conclusion

The authors successfully validated their reconstruction, matching 13 of 15 original UCI Adult variables. Their experiments with folktables show that local interventions (training state-specific models) often outperform national models in both accuracy and fairness. By "retiring" Adult and providing the folktables framework, the authors offer the research community a path toward more nuanced and context-aware fairness evaluations.

Summary updated

BibTeX
@inproceedings{ding2021retiring,
  author = {Frances Ding and Moritz Hardt and John Miller and Ludwig Schmidt},
  title = {{Retiring Adult: New Datasets for Fair Machine Learning}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=bYi_2708mKK},
}
Figure