The Misalignment of Click Signals and Search Relevance

This direction is merged into another direction: engagement-utility misalignment. Its last version stays readable and citable here.

Version 1 of 114 papers26 conceptsfrozen 2026-10-04

In the world of information retrieval and recommender systems, user clicks are the most abundant resource we have. They are cheap, real-time, and reflect actual behavior. For years, the working assumption has been that if we can build a model that perfectly predicts what a user will click on, we have built a perfect search engine.

However, a growing body of research suggests this assumption is fundamentally flawed. There is a persistent, structural gap between click-through rates (CTR) and true document relevance. Optimizing for the former does not guarantee the latter, and in many cases, it can actively degrade the user experience.

The Curiosity Gap

The primary reason clicks fail as a proxy for relevance is that a click is often an expression of curiosity rather than satisfaction. Joseph Noel and colleagues demonstrated this in New objective functions for social collaborative filtering, noting that users frequently engage with content only to find it unsatisfactory. Their study found that even a model with perfect click-prediction accuracy might only achieve a 66% accuracy rate in predicting whether a user actually liked the content.

This "curiosity gap" means that behavioral signals are inherently noisy. As Shuguang Han observed in PITT at TREC 2011 Session Track, click-based feedback tends to capture broad, session-level interests rather than the specific, granular information needs of a particular query. A user might click on a popular result because it looks familiar, not because it answers their immediate question.

Why Unbiased Learning Isn't Enough

To fix the noise in click data, researchers developed Unbiased Learning to Rank (ULTR). These methods use clever mathematics to "de-bias" clicks, accounting for the fact that users are more likely to click the first result simply because of its position.

But even when the bias is removed, the relevance problem remains. In a massive study of Baidu’s search logs, Romain Deffayet and Maarten de Rijke found that ULTR models often fail to outperform "naive" models in real-world scenarios. Their paper, Unbiased Learning to Rank Meets Reality: Lessons from Baidu's Large-Scale Search Dataset, highlights a "fundamental divergence": being better at predicting a click does not make a model better at finding what is objectively relevant according to expert human annotators.

The Disconnect Between Humans and Data

This divergence creates a "performance gap" between two different ways of measuring success:

  1. Implicit Feedback: Models trained on search logs and clicks.
  2. Explicit Judgment: Models evaluated against manual relevance labels provided by experts.

Xiangsheng Li and the THUIR2 team explored this in THUIR2 at NTCIR-16 Session Search (SS) Task, finding that models optimized for click logs often struggle when held up against human standards.

This is partly because human annotators and actual users are looking for different things. In E-commerce, for example, Shubhra Kanti Karmaker Santu found that human annotators are often poor at predicting what a user will actually buy because they lack context on the user's price sensitivity or brand loyalty. In On Application of Learning to Rank for E-Commerce Search, Santu argues that we should prioritize "order rate" over "click rate," as a purchase is a much stronger signal of utility than a mere click.

Bridging the Gap with Values

If clicks are unreliable, what should we optimize for? The research points toward two paths:

First, we can ground models in semantic relevance rather than just behavioral patterns. Sean MacAvaney showed in PACRR Gated Expansion for TREC CAR 2018 that while neural rankers are powerful, they still need to be carefully aligned with human judgments to avoid being led astray by proxy signals like document structure.

Second, we can move toward value-aware recommendation. Jonathan Stray and Alon Y. Halevy argue in Building Human Values into Recommender Systems that we must move beyond simple engagement metrics. By integrating non-behavioral signals—such as direct user feedback or expert ethical annotations—we can bridge the gap between what a user reflexively clicks and what they actually value.

The takeaway for the field is clear: a click is the start of a conversation, not the conclusion. To build truly effective systems, we must look past the engagement and toward the underlying utility.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_the_misalignment_of_click_signals_and_se_ce46bf87,
  title = {The Misalignment of Click Signals and Search Relevance},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/the-misalignment-of-click-signals-and-search-relevance/rdr_ce46bf872c0f438cbfa4af0a82481a7a},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_the_misalignment_of_click_signals_and_se_ce46bf87_v1,
  title = {The Misalignment of Click Signals and Search Relevance},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/the-misalignment-of-click-signals-and-search-relevance/rdr_ce46bf872c0f438cbfa4af0a82481a7a/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}