Online Information Retrieval Evaluation using the STELLA Framework

Timo Breuer, Narges Tavakolpoursaleh, Johann Schaible, Daniel Hienert, Philipp Schaer, Leyla Jael CastroarXiv 2022 · DOI · arXiv

TL;DR

The STELLA framework bridges the gap between academic information retrieval (IR) evaluation and real-world user behavior by creating a "living lab" infrastructure. It allows researchers to deploy experimental search and recommendation algorithms as Dockerized micro-services directly into live, production academic search engines, evaluating them via real-time user interactions.

What Problem Does This Paper Address?

Academic IR and recommendation systems traditionally rely on the "Cranfield paradigm"—an offline evaluation method using standardized document corpora and expert relevance assessments. While controlled and reproducible, this approach often fails to capture the messy reality of actual user needs and behaviors.

In contrast, commercial platforms continuously optimize their systems via online A/B testing and log analysis. The authors address this divide by providing an open infrastructure that enables academic researchers to conduct large-scale, real-time online evaluations with actual users, without disrupting the hosting production systems.

Methods

STELLA establishes a reproducible online evaluation environment using three core components:

  1. Micro-services: Participant experimental systems are containerized using Docker. They implement a REST-based web service with predefined endpoints for ranking or recommendation, allowing them to generate results on-the-fly.
  2. Multi-container Application (MCA): This bundles the experimental micro-services together. When a production system receives a query, the MCA uses Team-Draft-Interleaving to blend results from an experimental system with the production baseline. Interleaving prevents users from seeing completely broken results (a risk in pure A/B testing) and achieves statistical significance with less data.
  3. Central Server: Manages registration of participants, automates MCA updates, and hosts a dashboard providing visual analytics based on user interaction logs.
Figure 1: Infrastructure design illustrating how experimental systems can be contributed as Docker- based micro-services to online experiments with real users.

Datasets

Instead of relying on static, pre-compiled datasets, STELLA integrates directly into real-world production environments. For the evaluation presented in this paper (the CLEF 2021 LiLAS lab), it was integrated into two academic search platforms:

  • LIVIVO: Used for ad-hoc document ranking tasks with respect to a head query (life sciences domain).
  • GESIS Search: Used for dataset recommendation tasks with respect to a reference document (social sciences domain).

The evaluation data itself consists of real-time user click feedback collected over two 4-week rounds.

Key Insights and Ideas

  • On-the-fly vs. Pre-computed: Fully Dockerized systems allow for real-time, arbitrary query processing. This overcomes the limitations of traditional living labs, which restricted participants to pre-computed top-k lists for a fixed, historical set of queries.
  • The Value of Interleaving: By mixing experimental results with trusted baseline results, the framework protects the user experience while accelerating data collection for statistical significance tests.
  • Rethinking Click Metrics: Rather than treating all clicks equally, the authors implement a "Reward" measure. This metric weights user interactions based on where they click on the Search Engine Results Page (SERP), providing a more nuanced understanding of system performance.

Key Results

  • STELLA successfully served as the technological and methodological foundation for the CLEF 2021 Living Labs for Academic Search (LiLAS) lab.
  • The infrastructure successfully evaluated 9 experimental systems contributed by 3 participating groups across two 4-week rounds.
  • While most experimental systems did not outperform the production baselines, the granular click logging allowed researchers to identify distinct performance merits across different SERP elements.
  • The use of Docker significantly reduced deployment and migration overhead for site providers and organizers, making the experiments highly transparent and reproducible.

Limitations and Future Work

  • Data Sparsity: Some evaluation paths (especially for pre-computed recommendations) suffered from sparse click data, making definitive statistical conclusions difficult for those specific runs.
  • Baseline Transparency: The authors note a need to provide participants with more open and transparent baseline systems to build upon in future iterations.
  • Continuous Evaluation: Future work aims to transition from fixed-time evaluation "rounds" to a continuous, ongoing evaluation challenge model, freeing the living lab service from arbitrary time limits.

Summary updated

Abstract

Involving users in early phases of software development has become a common strategy as it enables developers to consider user needs from the beginning. Once a system is in production, new opportunities to observe, evaluate and learn from users emerge as more information becomes available. Gathering information from users to continuously evaluate their behavior is a common practice for commercial software, while the Cranfield paradigm remains the preferred option for Information Retrieval (IR) and recommendation systems in the academic world. Here we introduce the Infrastructures for Living Labs STELLA project which aims to create an evaluation infrastructure allowing experimental systems to run along production web-based academic search systems with real users. STELLA combines user interactions and log files analyses to enable large-scale A/B experiments for academic search.
BibTeX
@misc{breuer2022online,
  author = {Timo Breuer and Narges Tavakolpoursaleh and Johann Schaible and Daniel Hienert and Philipp Schaer and Leyla Jael Castro},
  title = {{Online Information Retrieval Evaluation using the STELLA Framework}},
  year = {2022},
  eprint = {2210.13202},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2210.13202},
}
Figure