This paper explores how to reliably evaluate Information Retrieval (IR) systems in "living labs"—real-world, small-scale search platforms—where user interaction data is often too sparse for traditional online metrics. The authors propose a validation framework that uses click models to generate synthetic usage data, allowing researchers to pre-screen experimental ranking algorithms before exposing real users to potentially poor search results.
TL;DR
The authors introduce a "Tester-based" approach to validate click models in data-sparse environments by checking if they can reproduce known system rankings. Their findings show that while simple models like DCTR are robust with as few as 20 sessions, complex models (DCM, SDBN) eventually provide more reliable simulations for interleaving experiments once more data is available.
What Problem Does This Paper Address?
Evaluating IR systems typically requires either expensive editorial relevance judgments (the "Cranfield paradigm") or massive amounts of user interaction data (A/B testing), which only large-scale search engines possess. "Living labs" offer a middle ground for smaller, domain-specific search services, but they face a "data sparsity" challenge: there isn't enough traffic to achieve statistical significance quickly.
Click models—probabilistic models that simulate how users interact with search results—could bridge this gap by generating synthetic data. However, it remains unclear how much data is needed to parameterize these models reliably. If a click model is trained on too little data, its simulations might be misleading, potentially leading researchers to deploy ineffective systems.
Methods
The authors employ a Tester-based validation framework. The core idea is simple: if a click model is a good proxy for real users, it should be able to correctly rank systems for which we already know the relative performance (e.g., we know a sophisticated BM25 ranker should outperform a random or null ranker).

They compare three click models:
- DCTR (Document-based Click-Through Rate): A simple model that assumes click probability depends only on a document's attractiveness at a specific query.
- DCM (Dependent Click Model): A more complex model that accounts for the user's persistence (the probability they continue scanning after a click).
- SDBN (Simplified Dynamic Bayesian Network): Similar to DCM but adds a "satisfaction" parameter specific to query-document pairs.
The models are evaluated using Log-Likelihood and Simulated Interleaving (specifically Team Draft Interleaving), measuring the correlation between the model's predicted ranking and a reference ranking using Kendall’s $\tau$.
Datasets
- TripClick: A large-scale dataset of logs from a biomedical search engine. The authors focused on the 50 most frequent queries to ensure a baseline of at least 100 sessions per query.
- TripJudge: A set of editorial relevance labels used as an external "ground truth" to validate their hypothesized reference rankings.
Key Insights and Ideas
- Complexity vs. Robustness: There is a clear trade-off. The simple DCTR model is remarkably robust; it can reproduce correct system rankings with very little data. In contrast, complex models like DCM and SDBN are "noisier" when data is sparse because they have more parameters to estimate, but they become superior simulators for interleaving experiments once they have sufficient data (around 100 sessions).
- System Diversity Matters: It is much easier for click models to distinguish between fundamentally different retrieval methods (e.g., BM25 vs. ranking by document length) than to distinguish between subtle variations of the same algorithm (e.g., different interpolation weights).
- Position Bias is a Double-Edged Sword: While complex models try to account for position bias (the tendency for users to click top results regardless of relevance), this extra complexity can actually hurt performance in extremely data-sparse settings compared to the position-agnostic DCTR.
Key Results
- DCTR Efficiency: With just 20 logged sessions for 50 queries, DCTR achieved a perfect Kendall’s $\tau$ of 1.0 in reproducing the lexical retrieval method (LRM) ranking.
- Interleaving Robustness: In simulated interleaving experiments, DCM and SDBN provided more stable and less error-prone estimates than DCTR once the number of sessions increased, suggesting they are better at capturing the nuances of user preference in head-to-head system comparisons.
- Transformer Validation: The authors applied their method to evaluate a state-of-the-art Transformer-based ranker (SPECTER). The models correctly identified that reranking at least the top 70 BM25 results significantly improved performance, demonstrating the practical utility of the framework for tuning modern IR pipelines.
Limitations and Future Work
- The "Unseen Document" Problem: Click models can only estimate the attractiveness of documents they have seen in historical logs. If a new experimental system brings up many previously unranked documents, the click model's estimates become less reliable.
- Inherent Biases: The authors acknowledge that the source logs contain biases from the original production system that are difficult to fully decouple.
- Future Directions: The next step is to validate these synthetic findings against actual live user behavior in a living lab setting to see if the "synthetic winners" truly translate to "real-world winners."
ACM J. Data Inform. Quality, Publication date: October 2023.
Summary updated