Debias Can Be Unreliable: Mitigating Bias in Evaluating Debiasing Recommendation

Chengbing Wang, Wentao Shi, Jizhi Zhang, Wenjie Wang, Hang Pan, Fuli Feng

TL;DR

This paper reveals that traditional evaluation methods for debiasing recommendation systems are unreliable when using randomly sampled user feedback data. They introduce a novel "Unbiased Recall Evaluation" (URE) scheme that accurately estimates true performance on fully observed data, enabling better model selection and development.

What Problem Does This Paper Address?

Recommendation systems often suffer from various biases, leading to inaccurate suggestions and potentially negative user experiences. To combat this, researchers develop "debiasing" techniques. The gold standard for evaluating these debiasing methods is a "fully-exposed" dataset, where user feedback is available for all items. However, such datasets are rare and expensive to collect. Consequently, most studies rely on "randomly-exposed" datasets, which contain feedback on a small, random subset of items. The problem is that traditional evaluation metrics, like Recall, calculated on these randomly-exposed datasets don't accurately reflect how well a debiasing model would perform on a fully-exposed dataset. This inconsistency can lead to misleading conclusions about the effectiveness of debiasing algorithms.

Methods

The core contribution of this paper is the Unbiased Recall Evaluation (URE) scheme. Unlike traditional methods that directly compute Recall on randomly-exposed data, URE leverages the model's predicted scores for all items. Here's how it works:

  1. Predict Scores: For each user, the model predicts scores for all candidate items.
  2. Rank Items: These items are then ranked based on the model's predicted scores.
  3. Identify Threshold: The (K+1)-th item in this ranked list is identified.
  4. Estimate Positive Ratio: Using the randomly-exposed dataset, the proportion of items with positive feedback that appear before this (K+1)-th item is calculated. This proportion serves as an unbiased estimate of the true Recall@K on a fully-exposed dataset.
  5. Average: This estimated positive ratio is averaged across all users to get the final URE score.

The authors provide theoretical guarantees (Theorem 2) that URE provides an unbiased estimate of the true Recall@K.

Datasets

The paper uses the following datasets:

  • Yahoo!R3: A dataset where a fully-exposed evaluation is not available.
  • KuaiRec: A dataset that provides both randomly-exposed (Drand) and fully-exposed (Dfull) data, crucial for validating the URE scheme.

Key Insights and Ideas

  • The "randomly-exposed" trap: Traditional evaluation on randomly-exposed datasets is fundamentally flawed because the Recall@K calculated on this sparse data has a weak correlation with the true Recall@K on fully-exposed data, especially for small values of K (which are often more important in practice).
  • K matters, a lot: The correlation between randomly-exposed and fully-exposed Recall@K is highly dependent on the value of K. A model that looks good with a small K on random data might perform poorly on full data, and vice-versa.
  • URE bridges the gap: By considering the model's full ranking and using the (K+1)-th item as a dynamic threshold, URE effectively "simulates" the behavior on a fully-exposed dataset, even when only randomly-exposed data is available.
  • Debiasing isn't always better: Re-evaluating existing debiasing methods using URE revealed that some methods, when evaluated correctly, actually perform worse than the original, non-debiased models for certain metrics (like Recall@5). This highlights the need for robust evaluation.

Key Results

  • Theoretical proof of URE's unbiasedness: The paper mathematically demonstrates that URE provides an unbiased estimate of Recall@K on fully-exposed data.
  • Empirical validation: Experiments show that URE's estimated Recall@K aligns closely with the true Recall@K on fully-exposed data (KuaiRec dataset), while traditional methods show significant discrepancies.
  • Improved model selection: URE allows for more reliable selection of debiasing models, as it accurately reflects their performance on the target (fully-exposed) scenario. For instance, Figure 3 shows that URE's estimated Recall@K has a much higher correlation with the true Recall@K than traditional methods, especially for small K.
  • Re-evaluation of debiasing methods: Table 1 shows that URE provides a more accurate picture of debiasing performance. For example, on KuaiRec, URE's Recall@5 for AutoDebias is 0.0323, which aligns with the true Recall@5 of 0.0550, whereas the traditional method yields a much lower 0.0237.

Limitations and Future Work

  • Availability of Drand: The URE scheme still requires a randomly-exposed dataset (Drand) to function. While more accessible than Dfull, its quality and representativeness are still important.
  • Computational Cost: While URE is designed to work with randomly-exposed data, it still requires the model to generate predictions for all items, which might be computationally intensive for very large item catalogs.
  • Focus on Recall: The paper primarily focuses on evaluating Recall. Future work could explore extending URE to other relevant evaluation metrics.
  • Further exploration of debiasing performance: The paper's re-evaluation suggests that some debiasing methods might need further refinement or entirely new approaches to truly improve performance across all metrics.

Summary updated

BibTeX
@inproceedings{wang2025debias,
  author = {Chengbing Wang and Wentao Shi and Jizhi Zhang and Wenjie Wang and Hang Pan and Fuli Feng},
  title = {{Debias Can Be Unreliable: Mitigating Bias in Evaluating Debiasing Recommendation}},
  booktitle = {The Web Conference},
  year = {2025},
  url = {https://openreview.net/forum?id=58S4HTvu0q},
}
Figure