TL;DR
The STELLA framework bridges the gap between academic information retrieval (IR) evaluation and real-world user behavior by creating a "living lab" infrastructure. It allows researchers to deploy experimental search and recommendation algorithms as Dockerized micro-services directly into live, production academic search engines, evaluating them via real-time user interactions.
What Problem Does This Paper Address?
Academic IR and recommendation systems traditionally rely on the "Cranfield paradigm"—an offline evaluation method using standardized document corpora and expert relevance assessments. While controlled and reproducible, this approach often fails to capture the messy reality of actual user needs and behaviors.
In contrast, commercial platforms continuously optimize their systems via online A/B testing and log analysis. The authors address this divide by providing an open infrastructure that enables academic researchers to conduct large-scale, real-time online evaluations with actual users, without disrupting the hosting production systems.
Methods
STELLA establishes a reproducible online evaluation environment using three core components:
- Micro-services: Participant experimental systems are containerized using Docker. They implement a REST-based web service with predefined endpoints for ranking or recommendation, allowing them to generate results on-the-fly.
- Multi-container Application (MCA): This bundles the experimental micro-services together. When a production system receives a query, the MCA uses Team-Draft-Interleaving to blend results from an experimental system with the production baseline. Interleaving prevents users from seeing completely broken results (a risk in pure A/B testing) and achieves statistical significance with less data.
- Central Server: Manages registration of participants, automates MCA updates, and hosts a dashboard providing visual analytics based on user interaction logs.
Datasets
Instead of relying on static, pre-compiled datasets, STELLA integrates directly into real-world production environments. For the evaluation presented in this paper (the CLEF 2021 LiLAS lab), it was integrated into two academic search platforms:
- LIVIVO: Used for ad-hoc document ranking tasks with respect to a head query (life sciences domain).
- GESIS Search: Used for dataset recommendation tasks with respect to a reference document (social sciences domain).
The evaluation data itself consists of real-time user click feedback collected over two 4-week rounds.
Key Insights and Ideas
- On-the-fly vs. Pre-computed: Fully Dockerized systems allow for real-time, arbitrary query processing. This overcomes the limitations of traditional living labs, which restricted participants to pre-computed top-k lists for a fixed, historical set of queries.
- The Value of Interleaving: By mixing experimental results with trusted baseline results, the framework protects the user experience while accelerating data collection for statistical significance tests.
- Rethinking Click Metrics: Rather than treating all clicks equally, the authors implement a "Reward" measure. This metric weights user interactions based on where they click on the Search Engine Results Page (SERP), providing a more nuanced understanding of system performance.
Key Results
- STELLA successfully served as the technological and methodological foundation for the CLEF 2021 Living Labs for Academic Search (LiLAS) lab.
- The infrastructure successfully evaluated 9 experimental systems contributed by 3 participating groups across two 4-week rounds.
- While most experimental systems did not outperform the production baselines, the granular click logging allowed researchers to identify distinct performance merits across different SERP elements.
- The use of Docker significantly reduced deployment and migration overhead for site providers and organizers, making the experiments highly transparent and reproducible.
Limitations and Future Work
- Data Sparsity: Some evaluation paths (especially for pre-computed recommendations) suffered from sparse click data, making definitive statistical conclusions difficult for those specific runs.
- Baseline Transparency: The authors note a need to provide participants with more open and transparent baseline systems to build upon in future iterations.
- Continuous Evaluation: Future work aims to transition from fixed-time evaluation "rounds" to a continuous, ongoing evaluation challenge model, freeing the living lab service from arbitrary time limits.
Summary updated