Debias Can Be Unreliable: Mitigating Bias in Evaluating Debiasing Recommendation

Chengbing Wang, Wentao Shi, Jizhi Zhang, Wenjie Wang, Hang Pan, Fuli Feng

TL;DR

Evaluating debiasing recommendation models using sparse, randomly-exposed datasets leads to highly inaccurate results. This paper introduces Unbiased Recall Evaluation (URE), a framework that accurately estimates true recommendation performance from sparse data by leveraging the model's full ranking behavior across the entire item catalog.

The Challenge of Sparse Evaluation

In recommender systems research, "gold standard" datasets where every user has rated every item are rare. Consequently, researchers rely on sparse, randomly-exposed datasets to evaluate debiasing techniques. However, this paper reveals a critical flaw: the standard "Recall@K" metric calculated on sparse data is a biased estimator of the true Recall on a fully-exposed dataset.

The "Small K" Trap and Evaluation Bias

The authors identify a phenomenon they call the "Small K Trap." They demonstrate that performance measured at a small rank (e.g., Recall@5) on a sparse dataset often correlates most strongly with performance at a much larger rank (e.g., Recall@500) on the full dataset. This "evaluation bias" means that models optimized for sparse benchmarks may fail to deliver the desired top-K performance in real-world applications.

Rank discrepancy in traditional evaluation: performance on sparse data (small K) correlates with much larger ranks on full data.

As shown in the figure above, the peak correlation ($K_{max}$) for a target rank on sparse data ($\bar{K}$) is shifted significantly to the right on the full dataset's rank scale ($K$). Even increasing the number of observed items per user ($\bar{N}$) does not fully resolve this discrepancy, highlighting a fundamental misalignment in traditional evaluation schemes.

Unbiased Recall Evaluation (URE)

To address this, the authors propose the Unbiased Recall Evaluation (URE) scheme. Unlike traditional methods that only consider items present in the test set, URE uses the model's predictions across the entire item catalog to "probe" its ranking behavior.

The URE Process: 1. Full Prediction: Score every item in the catalog for each user. 2. Global Ranking: Sort all items based on these scores. 3. Thresholding: Identify the score of the $(K+1)$-th item as a threshold. 4. Unbiased Estimation: Calculate the ratio of positive items from the sparse test set that exceed this threshold. 5. Averaging: Average this ratio across users to estimate true Recall@K.

The paper provides a rigorous mathematical proof (Theorem 2) demonstrating that URE is an unbiased estimator of the true Recall on a fully-exposed dataset.

Key Results and Insights

Validation on the KuaiRec dataset (a rare fully-observed dataset) confirms URE's accuracy. While traditional evaluation correlation drops below 0.6 for small $K$, URE maintains a Pearson correlation coefficient of > 0.9.

URE correctly aligns estimated performance with the intended rank K, achieving peak correlation at the target rank.

Furthermore, re-evaluating popular debiasing models (such as IPS, DR, and AutoDebias) on the Yahoo!R3 benchmark using URE revealed significant "rank reversals." Some advanced methods actually performed worse than simple Matrix Factorization when evaluated correctly, suggesting that previous benchmarks may have overestimated progress in the field due to evaluation bias.

Limitations and Future Work

While URE provides a robust solution for Recall, extending this logic to other ranking metrics like NDCG (Normalized Discounted Cumulative Gain) remains an open challenge. Additionally, URE assumes test data is "Missing at Random" (MAR) and requires higher computational effort, as it involves scoring all candidate items for every user during evaluation.

Summary updated

BibTeX
@inproceedings{wang2025debias,
  author = {Chengbing Wang and Wentao Shi and Jizhi Zhang and Wenjie Wang and Hang Pan and Fuli Feng},
  title = {{Debias Can Be Unreliable: Mitigating Bias in Evaluating Debiasing Recommendation}},
  booktitle = {The Web Conference},
  year = {2025},
  url = {https://openreview.net/forum?id=58S4HTvu0q},
}

Loading related content...

Figure