Abstract
Recommender systems often suffer from selection bias as users tend to rate their preferred items. The datasets collected under such conditions exhibit entries missing not at random (MNAR) and thus are not randomized-controlled trials representing the target population. To address this challenge, a doubly robust estimator and its enhanced variants have been proposed as they ensure unbiasedness when accurate imputed errors or predicted propensities are provided. However, we argue that existing estimators rely on miscalibrated imputed errors and propensity scores as they depend on rudimentary models for estimation. We provide theoretical insights into how miscalibrated imputation and propensity models may limit the effectiveness of doubly robust estimators and validate our theorems using real-world datasets. On this basis, we propose a Doubly Calibrated Estimator that involves the calibration of both the imputation and propensity models. To achieve this, we introduce calibration experts that consider different logit distributions across users. Moreover, we devise a tri-level joint learning framework, allowing the simultaneous optimization of calibration experts alongside prediction and imputation models. Through extensive experiments on real-world datasets, we demonstrate the superiority of the Doubly Calibrated Estimator in the context of debiased recommendation tasks.
Introduction
In modern recommender systems, the data available for training is rarely a representative sample of the general population. Instead, it is "Missing Not At Random" (MNAR). Users typically interact with and rate items they are already predisposed to like, leading to a significant selection bias. If a model is trained on this biased data without correction, it will likely fail to generalize to the broader item space, reinforcing "filter bubbles" and reducing the quality of discovery.
The Limits of Doubly Robust Estimators
To mitigate MNAR bias, researchers have turned to Doubly Robust (DR) estimators. These estimators attempt to bridge the gap between biased training data and the target population by combining two distinct strategies:
- Error Imputation: Predicting the likely error for unobserved user-item pairs to fill in the gaps of the missing data.
- Inverse Propensity Scoring (IPS): Weighting observed ratings by the inverse of the probability (propensity) that the interaction was observed in the first place.
The theoretical appeal of DR estimators lies in their "double robustness": the final recommendation model remains unbiased if either the imputation model or the propensity model is accurately specified. However, in practice, both models are often implemented using deep neural networks which, while powerful, are notoriously prone to miscalibration. Specifically, these models tend to be overconfident—predicting probabilities near 0 or 1 that do not reflect the true underlying frequencies. This overconfidence leads to high variance and significant bias, undermining the very robustness the estimators were designed to provide.
The Doubly Calibrated Estimator (DCE)
This paper introduces the Doubly Calibrated Estimator (DCE), a novel framework designed to address the miscalibration problem at its source. Rather than simply adding complexity to the recommendation model, DCE focuses on calibrating the internal components of the DR estimator to ensure they reflect reality.
Calibration Experts
Recognizing that user behavior is heterogeneous, the authors argue that a global calibration strategy is insufficient. DCE employs a Mixture-of-Experts (MoE) approach for calibration. Each "expert" utilizes a calibration function (such as Platt scaling) tailored to specific logit distributions. This allows the model to account for the fact that different users have different rating behaviors and scales.
Assignment Network
To determine which expert should handle which user, the framework includes an assignment network. Using a Gumbel-Softmax optimization, the model learns to group similar users together and assign them to the most appropriate calibration expert. This grouping mechanism allows the model to share knowledge across similar users, effectively mitigating the "cold-start" problem where individual users might have too little data for reliable individual calibration.
Tri-Level Joint Learning
A significant challenge in this architecture is the interdependency of the components. The authors propose a tri-level joint learning framework that simultaneously optimizes the primary prediction model, the imputation model, and the calibration experts. This ensures that as the predictive models evolve during training, the calibration experts adapt in real-time to maintain accuracy.
Theoretical and Empirical Validation
The paper provides a rigorous mathematical foundation, proving that the bias and variance of a DR estimator are directly bounded by the Calibration Error. By minimizing this error through the DCE framework, the researchers provide a mathematical guarantee of improved estimator stability.
Empirically, the model was tested on three benchmark datasets: Coat, Yahoo!R3, and KuaiRec. These datasets are unique because they contain "unbiased" test sets where users were asked to rate items randomly, providing a ground truth for debiasing performance.
Key Findings
- Accuracy Gains: The DCE-TDR variant achieved an 8.37% improvement in Mean Squared Error (MSE) on the Coat dataset compared to standard DR models.
- Ranking Performance: The model showed superior performance in ranking metrics such as NDCG and AUC, indicating that calibration helps the system better prioritize items for users rather than just predicting raw scores.
- Variance Reduction: Analysis of propensity scores confirmed that DCE prevents the "variance explosion" typically caused by the overconfident, near-zero propensity scores found in uncalibrated models.
Conclusion
The Doubly Calibrated Estimator represents a significant step forward in debiased recommendation. By identifying and fixing the "overconfidence" in propensity and imputation models, DCE restores the theoretical promises of doubly robust estimation. The framework is computationally efficient—adding less than 10% to training time—and is "orthogonal" to existing methods, suggesting it could be integrated into a wide variety of future recommendation architectures to improve performance in the face of biased data.
Summary updated