Validating Synthetic Usage Data in Living Lab Environments

Timo Breuer, Norbert Fuhr, Philipp SchaerarXiv 2023 · DOI · arXiv

This paper explores how to reliably evaluate Information Retrieval (IR) systems in "living labs"—real-world, small-scale search platforms—where user interaction data is often too sparse for traditional online metrics. The authors propose a validation framework that uses click models to generate synthetic usage data, allowing researchers to pre-screen experimental ranking algorithms before exposing real users to potentially poor search results.

TL;DR

The authors introduce a "Tester-based" approach to validate click models in data-sparse environments by checking if they can reproduce known system rankings. Their findings show that while simple models like DCTR are robust with as few as 20 sessions, complex models (DCM, SDBN) eventually provide more reliable simulations for interleaving experiments once more data is available.

What Problem Does This Paper Address?

Evaluating IR systems typically requires either expensive editorial relevance judgments (the "Cranfield paradigm") or massive amounts of user interaction data (A/B testing), which only large-scale search engines possess. "Living labs" offer a middle ground for smaller, domain-specific search services, but they face a "data sparsity" challenge: there isn't enough traffic to achieve statistical significance quickly.

Click models—probabilistic models that simulate how users interact with search results—could bridge this gap by generating synthetic data. However, it remains unclear how much data is needed to parameterize these models reliably. If a click model is trained on too little data, its simulations might be misleading, potentially leading researchers to deploy ineffective systems.

Methods

The authors employ a Tester-based validation framework. The core idea is simple: if a click model is a good proxy for real users, it should be able to correctly rank systems for which we already know the relative performance (e.g., we know a sophisticated BM25 ranker should outperform a random or null ranker).

![Fig.

  1. Click model evaluations based on system rankings and logged user interaction data from living labs.](/figures/art_ae21248439d0405ba1f67c07094d97c3)

They compare three click models:

  1. DCTR (Document-based Click-Through Rate): A simple model that assumes click probability depends only on a document's attractiveness at a specific query.
  2. DCM (Dependent Click Model): A more complex model that accounts for the user's persistence (the probability they continue scanning after a click).
  3. SDBN (Simplified Dynamic Bayesian Network): Similar to DCM but adds a "satisfaction" parameter specific to query-document pairs.

The models are evaluated using Log-Likelihood and Simulated Interleaving (specifically Team Draft Interleaving), measuring the correlation between the model's predicted ranking and a reference ranking using Kendall’s $\tau$.

Datasets

  • TripClick: A large-scale dataset of logs from a biomedical search engine. The authors focused on the 50 most frequent queries to ensure a baseline of at least 100 sessions per query.
  • TripJudge: A set of editorial relevance labels used as an external "ground truth" to validate their hypothesized reference rankings.

Key Insights and Ideas

  • Complexity vs. Robustness: There is a clear trade-off. The simple DCTR model is remarkably robust; it can reproduce correct system rankings with very little data. In contrast, complex models like DCM and SDBN are "noisier" when data is sparse because they have more parameters to estimate, but they become superior simulators for interleaving experiments once they have sufficient data (around 100 sessions).
  • System Diversity Matters: It is much easier for click models to distinguish between fundamentally different retrieval methods (e.g., BM25 vs. ranking by document length) than to distinguish between subtle variations of the same algorithm (e.g., different interpolation weights).
  • Position Bias is a Double-Edged Sword: While complex models try to account for position bias (the tendency for users to click top results regardless of relevance), this extra complexity can actually hurt performance in extremely data-sparse settings compared to the position-agnostic DCTR.

Key Results

  • DCTR Efficiency: With just 20 logged sessions for 50 queries, DCTR achieved a perfect Kendall’s $\tau$ of 1.0 in reproducing the lexical retrieval method (LRM) ranking.
  • Interleaving Robustness: In simulated interleaving experiments, DCM and SDBN provided more stable and less error-prone estimates than DCTR once the number of sessions increased, suggesting they are better at capturing the nuances of user preference in head-to-head system comparisons.
  • Transformer Validation: The authors applied their method to evaluate a state-of-the-art Transformer-based ranker (SPECTER). The models correctly identified that reranking at least the top 70 BM25 results significantly improved performance, demonstrating the practical utility of the framework for tuning modern IR pipelines.

Limitations and Future Work

  • The "Unseen Document" Problem: Click models can only estimate the attractiveness of documents they have seen in historical logs. If a new experimental system brings up many previously unranked documents, the click model's estimates become less reliable.
  • Inherent Biases: The authors acknowledge that the source logs contain biases from the original production system that are difficult to fully decouple.
  • Future Directions: The next step is to validate these synthetic findings against actual live user behavior in a living lab setting to see if the "synthetic winners" truly translate to "real-world winners."

ACM J. Data Inform. Quality, Publication date: October 2023.

Summary updated

Abstract

Evaluating retrieval performance without editorial relevance judgments is challenging, but instead, user interactions can be used as relevance signals. Living labs offer a way for small-scale platforms to validate information retrieval systems with real users. If enough user interaction data are available, click models can be parameterized from historical sessions to evaluate systems before exposing users to experimental rankings. However, interaction data are sparse in living labs, and little is studied about how click models can be validated for reliable user simulations when click data are available in moderate amounts. This work introduces an evaluation approach for validating synthetic usage data generated by click models in data-sparse human-in-the-loop environments like living labs. We ground our methodology on the click model's estimates about a system ranking compared to a reference ranking for which the relative performance is known. Our experiments compare different click models and their reliability and robustness as more session log data becomes available. In our setup, simple click models can reliably determine the relative system performance with already 20 logged sessions for 50 queries. In contrast, more complex click models require more session data for reliable estimates, but they are a better choice in simulated interleaving experiments when enough session data are available. While it is easier for click models to distinguish between more diverse systems, it is harder to reproduce the system ranking based on the same retrieval algorithm with different interpolation weights. Our setup is entirely open, and we share the code to reproduce the experiments.
BibTeX
@misc{breuer2023validating,
  author = {Timo Breuer and Norbert Fuhr and Philipp Schaer},
  title = {{Validating Synthetic Usage Data in Living Lab Environments}},
  year = {2023},
  eprint = {2310.07142},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2310.07142},
}
Figure