Abstract
Accurate estimates of examination bias are crucial for unbiased learning-to-rank from implicit feedback in search engines and recommender systems, since they enable the use of Inverse Propensity Score (IPS) weighting techniques to address selection biases and missing data. Unfortunately, existing examination-bias estimators are limited to the Position-Based Model (PBM), where the examination bias may only depend on the rank of the document. To overcome this limitation, we propose a Contextual Position-Based Model (CPBM) where the examination bias may also depend on a context vector describing the query and the user. Furthermore, we propose an effective estimator for the CPBM based on intervention harvesting. A key feature of the estimator is that it does not require disruptive interventions but merely exploits natural variation resulting from the use of multiple historic ranking functions. Real-world experiments on the ArXiv search engine and semi-synthetic experiments on the Yahoo Learning-To-Rank dataset demonstrate the superior effectiveness and robustness of the new approach.
Introduction
Learning-to-rank (LTR) systems for search engines and recommenders rely heavily on implicit feedback, primarily clicks. However, clicks are inherently biased by the position of results—a phenomenon known as examination bias. Users are more likely to examine and click top-ranked items regardless of their relevance. To correct this, Inverse Propensity Scoring (IPS) is used to weight clicks by the inverse of their examination probability.
Traditionally, this probability is estimated using a Position-Based Model (PBM), which assumes examination bias depends solely on rank. This paper argues that human behavior is context-dependent: a user's scanning pattern changes based on the query type or user intent. Furthermore, estimating these biases often requires disruptive "swap interventions" that degrade user experience. This work introduces the Contextual Position-Based Model (CPBM) and an "AllPairs" estimator that harvests natural variations in historical logs to learn these biases without active randomization.
Methods: The CPBM and Intervention Harvesting
The authors propose two key innovations to move beyond the limitations of standard position-based models:
- Contextual Position-Based Model (CPBM): Unlike the PBM, the CPBM models the propensity $P(E=1 | k, x)$ as a function of both rank $k$ and a context vector $x$. This context can include query features (e.g., query length, use of boolean operators) and user features.
- Intervention Harvesting: Instead of performing new, disruptive experiments, the AllPairs estimator exploits "implicit interventions" found in historical logs. If a search engine runs an A/B test with two different ranking algorithms, Algorithm A might place Document 1 at rank 1, while Algorithm B places it at rank 3. This natural variation acts as a free intervention, controlling for the document's actual relevance.
The CPBM is implemented using a neural network architecture. The AllPairs estimator trains two Multi-Layer Perceptrons (MLPs): one for context-dependent propensity and another for average relevance. This dual-model approach allows the system to learn how average relevance changes with context, making the optimization task significantly easier than learning exact relevance for every query-document pair.
Experimental Evaluation
The approach was validated using real-world data from the ArXiv search engine and semi-synthetic experiments on the Yahoo Learning-To-Rank dataset.
- Context Matters: The ArXiv experiments proved that propensity curves vary significantly by query type. "Complex" queries (using boolean operators like AND/OR) showed a much steeper drop-off in examination at lower ranks compared to "simple" queries.
- Accuracy and Efficiency: On the Yahoo LTR dataset, the CPBM reduced the relative error of propensity estimates by 64.6% compared to the standard PBM baseline. The model's error plummeted and stabilized after only about 5,700 training queries, demonstrating high data efficiency.
- Downstream Ranking Performance: When the CPBM's propensity scores were used in an unbiased LTR algorithm (Propensity SVM-Rank), the resulting ranking performance closely matched the theoretical gold standard trained with perfect, oracle propensities.
Conclusion
The CPBM and intervention harvesting provide a robust, non-disruptive way to estimate complex, context-dependent examination biases. By leveraging historical policy switches, systems can learn accurate propensity models without degrading the user experience. Future work could explore Recurrent Neural Networks (RNNs) to better model the sequential nature of user scanning behavior or integrate rank position directly as an input feature.
Summary updated