Counterfactual Inference for Unbiased Ranking

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 270 papers82 conceptsfrozen 2026-10-04

Most search and recommendation engines are trained on "implicit feedback"—the clicks, views, and purchases generated by users every day. While this data is abundant, it is fundamentally dishonest. A user cannot click an item they never saw, and they are far more likely to click the first result simply because it is at the top, regardless of its actual quality. This phenomenon, known as position bias, creates a "rich-get-richer" feedback loop that prevents systems from discovering truly relevant content.

To solve this, researchers have reframed ranking not as a simple prediction task, but as a problem of counterfactual inference. By treating the act of showing an item as a "treatment" in a clinical trial, we can mathematically "undo" the bias of the past to learn what users actually want.

The Recommendation as a Treatment

In a randomized controlled trial, doctors assign treatments to patients randomly to observe the true effect. In a search engine, the "treatment" is the position or exposure an item receives. However, historical logs are not random; they are "observational data" where the previous system decided which items to show.

Tobias Schnabel, Adith Swaminathan, and Thorsten Joachims formalized this in Recommendations as Treatments. They argued that if we know the probability that an item was shown (the propensity), we can re-weight our observations to simulate a randomized experiment.

The intuition is simple: if an item at rank 10 is clicked, it is a much stronger signal of relevance than a click at rank 1. Why? Because the probability of a user even looking at rank 10 is very low. To account for this, we weight the interaction by the Inverse Propensity Score (IPS).

Correcting the Loss Function

In standard machine learning, we minimize a loss function (like mean squared error) over all observed data. In an unbiased setting, we modify this objective. As Thorsten Joachims demonstrated with Propensity SVM-Rank in Unbiased Learning-to-Rank with Biased Feedback, each click is divided by its propensity $P(\text{examination})$.

$$ \text{Unbiased Risk} \approx \sum_{\text{clicks}} \frac{\text{Loss}(\text{item})}{\text{Propensity}(\text{position})} $$

This weighting ensures that the model focuses more on the "surprising" clicks at lower positions. If the propensities are accurate, this estimator is mathematically guaranteed to converge to the same model you would have built if you had perfect, unbiased labels from human experts.

From Points to Pairs

While early work focused on individual items (pointwise), modern ranking often relies on comparing pairs of items. Ziniu Hu and colleagues extended the IPS framework to these more complex models in A Novel Algorithm for Unbiased Learning to Rank.

Their Unbiased LambdaMART scales the loss of a document pair by the inverse propensity of both positions. This allows the model to learn from the relative order of items while still accounting for the fact that the user might not have seen the second item in the pair. Alexey Kurennoy further refined this in A General Framework for Pairwise Unbiased Learning to Rank, dropping the assumption that users look at items independently and allowing for more realistic browsing behaviors, like scanning a grid.

Beyond Position: Selection and Saturation

As the field matured, researchers realized that position is not the only source of bias.

  1. Selection Bias: Users can't click what isn't there. Zohreh Ovaisi addressed this in Correcting for Selection Bias in Learning-to-rank Systems by adapting the Heckman two-stage method from econometrics, which accounts for the "missingness" of unranked documents.
  2. Relevance Saturation: Keping Bi identified a "query-level" bias in Unbiased Learning to Rank with Query-Level Click Propensity Estimation. Even if multiple items are relevant, a user might stop clicking once their information need is satisfied. Their DualIPW mechanism weights samples by both their physical rank and the overall likelihood of any click occurring in that session.
  3. Multi-Stage Pipelines: Real-world systems use a "candidate generator" followed by a "re-ranker." Shashank Gupta and Maarten de Rijke showed in Towards Two-Stage Counterfactual Learning to Rank that propensities must be calculated cumulatively across both stages to avoid a distribution shift when one part of the system is updated.

Why It Works: The Power of the Counterfactual

The core strength of this research direction is its ability to turn "trash" (biased logs) into "gold" (unbiased signals) without requiring expensive manual labeling or risky real-time experiments. By explicitly modeling the user's observation process—whether through simple position curves or complex neural networks like the Self-adaptive Attention Module proposed by Jiabin Liu—we can build systems that are not just better at predicting clicks, but better at understanding true human preference.

Papers in this version

Showing 60 of 70 papers.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 108 papers
  2. Version 1 · 2026-10-04 · 70 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_inverse_propensity_weighting_cac29c66,
  title = {inverse propensity weighting},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/inverse-propensity-weighting/rdr_cac29c66a45543ef911eb0d12ea080d2},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_counterfactual_inference_for_unbiased_ra_cac29c66_v1,
  title = {Counterfactual Inference for Unbiased Ranking},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/counterfactual-inference-for-unbiased-ranking/rdr_cac29c66a45543ef911eb0d12ea080d2/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}