In search and recommendation systems, a click is the most common signal of success. However, a click is a noisy proxy for relevance. If a user doesn't click on the tenth result, is it because the result was bad, or because they never even looked at it? Conversely, if they click the first result, is it because it was perfect, or simply because it was the first thing they saw?
This fundamental ambiguity is known as position bias. If we train models directly on raw click data, they learn to favor items at the top of the list regardless of quality, creating a feedback loop that reinforces existing biases. To solve this, researchers have developed a suite of techniques to "de-bias" implicit feedback by modeling the hidden process of user examination.
The Examination Hypothesis
The core intuition behind this research is the Examination Hypothesis. It posits that a click ($C$) occurs only if two conditions are met: the user must examine the item ($E$) and the item must be relevant ($R$). Mathematically, this is expressed as:
$$P(C=1) = P(E=1) \times P(R=1)$$
The goal of Unbiased Learning to Rank (ULTR) is to isolate $P(R=1)$—the true relevance—by estimating and dividing out $P(E=1)$, often called the propensity.
Early models assumed that $P(E)$ depended solely on the rank of the item. However, Aman Agarwal and Thorsten Joachims demonstrated in Consistent Position Bias Estimation without Online Interventions for Learning-to-Rank that this is too simplistic. They showed that examination probability is influenced by observable context, such as the user interface (UI) layout or the category of the query. By incorporating these contextual features, we can more accurately estimate the likelihood that a user actually saw a specific result.
From Rank to Contextual Propensity
The shift from simple position-based models to contextual models allows systems to handle more complex user behaviors. Maosong Sun, Qingyun Wu, and their colleagues addressed this in the bandit setting with the E-C Bandit framework in Bandit Learning with Implicit Feedback.
Instead of treating examination as a fixed constant for each rank, they model it as a function of diverse contextual features—including user demographics and item metadata. Using Variational Bayesian inference, the E-C Bandit treats examination as a latent variable, allowing the system to distinguish between a user "ignoring" an item and "disliking" it. This is particularly effective in dynamic environments like MOOC platforms, where user attention varies wildly based on the context of the course they are viewing.
The Influence of the List Context
Users do not evaluate search results in a vacuum; they evaluate them relative to the other items on the page. Keping Bi and Jiafeng Guo have pushed the field toward listwise propensity modeling.
In Contextual Dual Learning Algorithm with Listwise Distillation for Unbiased Learning to Rank, they argue that the probability of examining a result is conditioned on the visual and topical context of the entire list. For example, if the first three results are highly attractive but irrelevant "clickbait," a user might get frustrated and stop looking further down, even if the fourth result is perfect. Their CDLA-LD framework uses a Transformer-based teacher model to capture these complex interactions across the whole list, distilling that knowledge into a student model that can rank items more accurately.
Relevance Saturation and Query Intent
Even if a user examines a relevant result, they might not click it if they have already found what they need. This is known as Relevance Saturation Bias.
In Unbiased Learning to Rank with Query-Level Click Propensity Estimation, Keping Bi and Dawei Yin introduce DualIPW, which accounts for both position-level and query-level propensity. The intuition is that click-through rates are influenced by the overall quality of the session. If a query is very specific and the first result is a perfect "featured snippet," the user is unlikely to click anything else, regardless of how relevant the subsequent links are.
Furthermore, the type of query matters. Linjun Shou and Daxin Jiang highlight in Mining Implicit Relevance Feedback from User Behavior for Web Question Answering that user behavior differs between "factoid" questions (where a quick glance suffices) and "how-to" questions (which require long dwell times). By incorporating behavioral signals like dwell time and re-query patterns, they show that we can mine much richer relevance signals than simple clicks alone.
Summary of Intuition
To build an unbiased ranker, we must stop treating clicks as ground-truth labels and start treating them as the result of a multi-stage psychological process:
- Contextual Examination: Does the UI, the rank, and the surrounding content make the user look at this item?
- Information Need: Is the user still looking, or has their need already been saturated by a previous result?
- True Relevance: Given that they looked and still need an answer, is this specific item helpful?
By modeling the first two stages as "propensities," we can mathematically re-weight the observed clicks to uncover the true relevance hidden beneath the surface of user behavior.