On the Fairness of Randomized Trials for Recommendation With Heterogeneous Demographics and Beyond

Xi ChenarXiv 2020 · arXiv

TL;DR

This paper tackles the challenge of ensuring fairness in recommendation systems when user data is "missing not at random" (MNAR), a common issue that biases standard machine learning models. They propose a novel Counterfactual Robust Risk Minimization (CRRM) framework that achieves fairness and accuracy without relying on expensive randomized trials, a common but costly debiasing method.

What Problem Does This Paper Address?

Recommendation systems, while aiming for high average performance, often exhibit unfairness towards minority user groups. This disparity can lead to a negative feedback loop: minority users receive poorer recommendations, become less engaged, and provide less data, further exacerbating the bias. A key challenge is that user interaction data is often "missing not at random" (MNAR). This means the observed data (e.g., clicks) is not a random sample of all possible interactions; users are more likely to interact with items they like, leading to biased estimates of true user preferences.

Traditional methods to correct MNAR bias include running small Randomized Controlled Trials (RCTs) to collect unbiased samples. However, RCTs are costly and may not adequately address fairness for heterogeneous user demographics. This paper aims to develop a method that is both fair and accurate, even with MNAR data and diverse user populations, without the need for RCTs.

Methods

The core contribution is the Counterfactual Robust Risk Minimization (CRRM) framework. This approach aims to balance two objectives:

  • Empirical Risk Minimization: Standardly minimizing prediction errors on observed data.
  • Counterfactual Robust Risk Minimization: Introducing a novel "robust risk" term that penalizes discrepancies between the model's predictions for observed data and slightly perturbed counterfactual data. This encourages the model to be robust to variations in user preferences, even those not directly observed.

The CRRM framework is designed to be "RCTs-free," meaning it doesn't require collecting additional unbiased data through randomized trials. It achieves this by incorporating a counterfactual component directly into the optimization objective. The paper also theoretically analyzes the fairness of existing RCT-based methods (like Inverse Propensity Score - IPS) under heterogeneous demographics, highlighting their limitations.

Datasets

  • Synthetic Tasks: Used to demonstrate the framework's effectiveness and analyze its behavior under controlled conditions, including homogeneous and heterogeneous demographics.
  • Yahoo! R3 Data Set: A user-song rating dataset used for evaluating the proposed method on real-world data.
  • Coat Shopping Data Set: Another real-world dataset used for evaluation.

Key Insights and Ideas

  • MNAR data exacerbates fairness issues: The paper demonstrates that MNAR data, common in recommendation logs, can worsen utility disparities for minority groups over time.
  • RCTs are not a silver bullet for fairness: While RCTs can correct MNAR bias, their effectiveness for fairness, especially with heterogeneous user demographics, is questioned. The paper shows that averaging RCT results can still lead to unfairness for specific groups.
  • Counterfactual robustness is key to fairness: The CRRM framework's core idea is that by making predictions robust to small, counterfactual perturbations, the model becomes less sensitive to the biases introduced by MNAR data and more equitable across different user groups.
  • Gini Index as a fairness metric: The paper adopts the Gini index, a measure from economics, to quantify utility disparity among users, providing a concrete way to evaluate fairness.

Key Results

  • The proposed CRRM framework significantly improves both fairness (measured by Gini index) and effectiveness (measured by global utility and accuracy metrics like MSE/AUC) compared to baseline methods (ERM, IPS-based MF) on both synthetic and real-world datasets.
  • On real-world datasets (Yahoo! R3 and Coat), CRRM achieved the best performance across fairness and effectiveness metrics, outperforming naive matrix factorization and various IPS-based methods.
  • The theoretical analysis shows that CRRM provides a generalization error bound that explicitly balances fairness and effectiveness, a trade-off not present in standard ERM.

Limitations and Future Work

  • Computational Cost: While CRRM is RCTs-free, the counterfactual robust risk minimization can still be computationally intensive. The paper mentions that random sampling can mitigate this for large datasets.
  • Hyperparameter Tuning: The performance of CRRM can be sensitive to hyperparameter choices, although the authors note that few parameters require fine-tuning in practice.
  • Future Work: The authors suggest exploring cutting-edge sampling approaches (beyond random sampling) for further improvement and adapting CRRM to other recommendation models like deep factorization machines and graph neural networks.

Summary updated

BibTeX
@misc{chen2020fairness,
  author = {Xi Chen},
  title = {{On the Fairness of Randomized Trials for Recommendation With Heterogeneous Demographics and Beyond}},
  year = {2020},
  eprint = {2001.09328},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2001.09328},
}
Figure