Weak Supervision from User Interaction Logs

30 concepts23 papers1 proposal

Modern deep learning models are notoriously data-hungry. Whether training a neural information retrieval system or aligning a generative model, performance scales directly with the volume of annotated training data. However, relying on expert human judges to manually label query-document relevance or rank candidate outputs creates a severe scalability bottleneck, as discussed by the authors of Structured Learning of Two-Level Dynamic Rankings and Generative Input: Towards Next-Generation Input Methods Paradigm. Manual annotation is not only prohibitively expensive and time-consuming, but it also struggles to capture the messy, dynamic nature of real-world user intents (Learning Query and Image Similarities with Ranking Canonical Correlation Analysis).

To bypass this bottleneck, a powerful paradigm has emerged: leveraging weak supervision from user interaction logs. Instead of asking humans to explicitly score relevance, researchers use the implicit feedback embedded in daily system interactions—such as search engine click logs, document access records, and user-generated content—as a scalable proxy for ground-truth labels.

Transforming Clicks into Relevance Proxies

The fundamental intuition behind this approach is that user behavior, when aggregated over millions of sessions, reflects underlying preferences. For instance, if users searching for a specific query frequently click a particular document, that document is highly likely to be relevant. In ORCAS: 20 Million Clicked Query-Document Pairs for Analyzing Search, Campos et al. demonstrated that aggregated click data correlates strongly with traditional expert human relevance metrics, providing a massive bridge between industrial scale and academic research.

This implicit feedback can be mined in several ways to generate training signals:

  • Pairwise Preferences: Rather than treating a click as an absolute score, models often learn from relative preferences. If a user clicks on result A but skips result B positioned above it, it implies a pairwise preference. In Learning to Explain Entity Relationships by Pairwise Ranking with Convolutional Neural Networks, Huang et al. utilized search engine clickthrough logs to derive pairwise preferences for ranking descriptive text, allowing a convolutional neural network to learn entity relationship explanations without manual templates.
  • Co-click Relations: Two distinct queries can be inferred as semantically similar if they frequently lead users to click on the exact same set of documents. De Bona et al. leveraged this "co-click" mechanism in Learning Dense Models of Query Similarity from User Click Logs to generate billions of weakly supervised query pairs, training dense models to rank query rewrites effectively.
  • Metric Learning and Triplet Mining: In cross-modal settings, user clicks can bridge the gap between different domains. The author of Visual Search at Alibaba used interaction logs for triplet mining, treating a user's clicked search result as a positive match and non-clicked items as hard negatives. This allowed a triplet loss function to align messy user-captured queries with professional product images without requiring manual bounding box annotations.

Overcoming Noise with Probabilistic Click Models

While interaction logs offer an almost infinite stream of data, they are notoriously noisy. A raw click does not automatically equal relevance; it is heavily influenced by position bias (users are more likely to click top results simply because they see them first) and presentation bias.

To extract a clean training signal, researchers employ probabilistic click models. As highlighted by Zheng et al. in THUIR at NTCIR-13 WWW Task and by Mao et al. in Overview of the NTCIR-14 We Want Web Task, these models mathematically isolate user attention and position biases from raw log data.

By filtering out these confounding variables, probabilistic browsing models transform raw clicks into high-quality, unbiased relevance signals, a technique also leveraged in LOVBench: Ontology Ranking Benchmark by Kolbe et al. to evaluate semantic data ranking. The massive volume of the resulting weakly supervised data ultimately compensates for the lack of precision in individual labels, enabling data-hungry neural ranking models to achieve performance levels comparable to those trained on pristine human judgments (THUIR at NTCIR-13 WWW Task; Overview of the NTCIR-15 We Want Web with CENTRE (WWW-3) Task).

The utility of behavior-driven weak supervision extends far beyond traditional web search engines:

  • Image Annotation: Li et al. demonstrated in Renmin University of China at ImageCLEF 2014 Scalable Concept Image Annotation that user click counts from search engine results can serve as a scalable proxy to train automated image tagging models on millions of uncurated web images, bypassing expert annotation costs entirely.
  • Domain-Specific Retrieval: In specialized environments like healthcare, Jiang et al. showed in Conceptualizing Machine Learning for Dynamic Information Retrieval of Electronic Health Record Notes that Electronic Health Record (EHR) audit logs—specifically document access records—can serve as a powerful proxy for relevance, allowing models to proactively suggest historical medical notes to clinicians without requiring manual expert labeling.
  • Large Language Model Alignment: Moving beyond traditional information retrieval, Tan et al. introduced a framework in Aligning Large Language Models with Implicit Preferences from User-Generated Content to align Large Language Models (LLMs) using the implicit preferences buried within massive, unlabelled web corpora like Reddit and Amazon reviews. The authors demonstrated that models aligned via these implicit behavioral signals can match or exceed the performance of models trained on expensive, human-curated preference datasets.

Why It Works: The Power of Scale and Surrogacy

Why does weak supervision from interaction logs work so well despite its inherent noise?

First, it relies on the law of large numbers. While an individual user's click might be accidental or misguided, the aggregate behavior of millions of users reveals robust statistical patterns. Integrating massive click-log datasets into the training pipeline provides a high-volume representational strategy that captures diverse query-document relationships, as detailed in TREC Deep Learning Track: Reusable Test Collections in the Large Data Regime by Yilmaz et al.

Second, it enables surrogate bootstrapping. When starting from scratch, the rankings of an established system can be used as surrogate labels to bootstrap a model, which can later be fine-tuned with smaller, high-quality datasets, an approach utilized by Craswell et al. in Microsoft Research at TREC 2009: Web and Relevance Feedback Track.

Finally, interaction logs capture actual user intent and satisfaction in the wild. Human annotators in a lab setting often guess what a user wants, whereas interaction logs record genuine human decisions, bridging the domain gap between messy user-generated queries and reference targets.

Key Foundational Work

Learning Dense Models of Query Similarity from User Click Logs (2010) — contributed 2 concept elements to this direction.
Using Word-Sense Disambiguation Methods to Classify Web Queries by Intent (2009) — contributed 1 concept elements to this direction.
Microsoft Research at TREC 2009: Web and Relevance Feedback Track (2009) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_co_click_weak_supervision_1bc08379,
  title = {co-click weak supervision},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/co-click-weak-supervision/rdr_1bc08379037c4bdb924d266969dc42d4},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_co_click_weak_supervision_1bc08379_v1,
  title = {co-click weak supervision},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/co-click-weak-supervision/rdr_1bc08379037c4bdb924d266969dc42d4/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake