Abstract
Counterfactual learning for dealing with missing-not-at-random data (MNAR) is an intriguing topic in the recommendation literature since MNAR data are ubiquitous in modern recommender systems. Missing-at-random (MAR) data, namely randomized controlled trials (RCTs), are usually required by most previous counterfactual learning methods for debiasing learning. However, the execution of RCTs is extraordinarily expensive in practice. To circumvent the use of RCTs, we build an information-theoretic counterfactual variational information bottleneck (CVIB), as an alternative for debiasing learning without RCTs. By separating the task-aware mutual information term in the original information bottleneck Lagrangian into factual and counterfactual parts, we derive a contrastive information loss and an additional output confidence penalty, which facilitates balanced learning between the factual and counterfactual domains. Empirical evaluation on real-world datasets shows that our CVIB significantly enhances both shallow and deep models, which sheds light on counterfactual learning in recommendation that goes beyond RCTs.
Introduction: The Challenge of MNAR Feedback
Modern recommender systems are plagued by selection bias, specifically Missing-Not-At-Random (MNAR) feedback. Users typically interact with items they already prefer, leading to a dataset dominated by "factual" observations. Standard models trained on this data develop a significant blind spot, overestimating preferences for certain items while ignoring the vast "counterfactual" space of unobserved items.
Traditionally, researchers address this by conducting Randomized Controlled Trials (RCTs) to obtain unbiased Missing-At-Random (MAR) data. However, RCTs are prohibitively expensive, potentially degrade user experience, and often fail to cover the entire item space. This research proposes a method to debias recommendation models without the need for any RCT training data.
Methodology: Counterfactual Variational Information Bottleneck (CVIB)
The proposed CVIB framework adapts the classical Information Bottleneck (IB) principle to a counterfactual setting. The IB principle seeks a representation $z$ that balances two competing goals: retaining enough information to predict the target $y$ (sufficiency) while compressing the representation to discard noise, such as the bias inherent in the logging policy (minimality).
The CVIB innovation lies in splitting the learning process into Factual ($O=1$) and Counterfactual ($O=0$) domains. The derived objective function includes:
- Sufficiency Term: Standard supervised learning on observed factual data.
- Contrastive Balancing Term: Encourages the model to be "equally informative" across both domains, forcing the discovery of features that generalize beyond observed interactions.
- Confidence Penalty: Prevents overconfidence in factual predictions, maintaining openness to counterfactual possibilities.
- Minimality Term: Uses Variational Inference (KL-divergence) to ensure embeddings do not simply memorize biased logging policies.
Experimental Evaluation
The authors evaluated CVIB on two benchmark datasets: Yahoo! R3 (over 300,000 ratings) and Coat Shopping (290 users, 300 items). Both datasets include small MAR test sets for unbiased evaluation. CVIB was compared against established baselines like Inverse Propensity Scoring (IPS) and Doubly Robust (DR) methods.
Key Results
- Superior Performance: CVIB consistently outperformed all baselines. On the Yahoo! dataset, it achieved an AUC of 0.7198 (compared to the next best at 0.6892) and an nDCG@10 of 0.820.
- Model Agnostic: The framework successfully enhanced both shallow models (Matrix Factorization) and deep models (Neural Collaborative Filtering).
- Ranking Efficacy: While CVIB occasionally showed higher Mean Squared Error (MSE), it consistently excelled in ranking metrics (AUC/nDCG), which are more critical for real-world recommendation quality.
Conclusion and Future Work
CVIB demonstrates that information-theoretic constraints can effectively debias models without requiring expensive RCT data. By optimizing the flow of information and ignoring policy bias, the model learns more fundamental user preferences. Future work may address hyperparameter sensitivity and the computational complexity of calculating contrastive terms over the entire counterfactual space.
Summary updated