TL;DR
FairJob is a new large-scale, real-world dataset designed to benchmark algorithmic fairness in the high-stakes domain of job advertising. It provides over 1 million rows of anonymized data, a novel behavior-based gender proxy, and a new utility metric to help researchers navigate the complex trade-offs between recommendation accuracy and social equity.
What Problem Does This Paper Address?
Algorithmic fairness research often suffers from a "reality gap." Most benchmarks rely on small, decades-old datasets like Adult or COMPAS that don't reflect the complexity of modern online systems. Meanwhile, in high-impact industries like advertising—which acts as a gatekeeper to employment, housing, and credit—real-world data is usually locked away due to privacy regulations (like GDPR) and business confidentiality.
Furthermore, practitioners rarely have access to "ground truth" protected attributes (like a user's actual gender) because collecting such sensitive data is often legally restricted or technically impossible. This paper introduces a realistic, large-scale resource that allows researchers to study these challenges—specifically the "Fairness-Utility" trade-off—in a privacy-compliant way.
Methods
The authors release a dataset of ~1.07 million rows collected from a 5-month job advertising campaign. To make this data public while protecting privacy, they used pseudonymization and random projections to mask feature values while preserving their predictive power.
Key technical contributions include:
- Causal Framework for Fairness: The authors model the relationship between observable attributes ($X$), protected attributes ($A$), and outcomes ($Y$) using causal graphs. They explore three regimes: direct dependence, fairness through unawareness (hiding $A$), and fairness via zero mutual information (Figure 2).
- Gender Proxy Derivation: Since true gender is unavailable, they derive a proxy based on user behavior—specifically, the dominant gender associated with the products a user interacts with.
- Selection Bias Correction: They identify that unfairness often starts before the recommendation engine, at the campaign selection stage. They propose a new Product-rank utility metric ($\tilde{U}$) that uses population-level weights to correct for this selection bias, ensuring that "fairness" isn't just measured within a potentially biased subset of users.
- Bias Mitigation Benchmarking: They evaluate standard models (XGBoost and Logistic Regression) using an in-processing fairness penalty that allows researchers to tune the importance of fairness relative to predictive accuracy.
Datasets
The primary contribution is the FairJob dataset:
- Scale: 1,072,226 rows and 60 features (20 categorical, 39 numerical).
- Task: Binary classification (predicting if a user will click a job ad).
- Protected Attribute: A binary gender proxy.
- Context: Job seniority (binary label for "senior" vs "assistant" roles).
The authors compare FairJob to other common tabular datasets (Table 9 & 10), noting that it is significantly larger and more feature-rich than traditional fairness benchmarks like Adult or Diabetes.
Key Insights and Ideas
- Unawareness is Not Fairness: Simply removing the gender attribute from training data (Figure 2b) is insufficient. Models easily learn to reconstruct the protected attribute from other correlated features, leading to "indirect" unfairness.
- The Hidden Impact of Selection Bias: If women are less likely to be targeted for a managerial job campaign in the first place, a "fair" recommendation engine within that campaign still results in population-level disparities. The proposed $\tilde{U}$ metric is a crucial tool for auditing this "market bias."
- Tabular Data is Under-Researched: While fairness in Computer Vision and NLP is booming, tabular data—the backbone of finance, hiring, and healthcare—lacks large-scale benchmarks. FairJob aims to fill this specific gap.
Key Results
The authors benchmarked several models to establish a baseline for the community:
- XGBoost Performance: An "unfair" XGBoost model achieved an AUC of 0.762, but at the cost of higher Demographic Parity (DP) disparity. The "unaware" version slightly improved fairness but dropped in predictive power (Table 1).
- Fairness-Utility Trade-off: By varying the fairness penalty ($\lambda$), the authors demonstrated that researchers can precisely control the balance between accuracy (NLLH) and fairness (DP). Interestingly, for certain penalty ranges, the "fair utility" metric ($\tilde{U}$) actually showed slight improvements even as standard accuracy metrics degraded (Figure 6).
Limitations and Future Work
The authors acknowledge several real-world constraints:
- Proxy Noise: The gender proxy is a behavioral estimate, not ground truth, and gender is treated as binary for simplicity—a limitation in capturing the full spectrum of identity.
- Subjectivity: The "seniority" label was derived from job titles, which can be subjective or noisy.
- Historical Bias: The dataset contains clicks from a system that may have already been biased, potentially baking historical inequalities into the "ground truth" labels.
Future work involves exploring more sophisticated deep learning models (like Tabular Transformers) and more complex bias mitigation techniques on this large-scale data.
Summary updated