The Information Retrieval Experiment Platform (Extended Abstract)

Maik Fröbe, Jan Heinrich Reimer, Sean MacAvaney, Niklas Deckers, Simon Reich, Janek Bevendorff, Benno Stein, Matthias Hagen, Martin PotthastIJCAI 2024

Abstract

TIREx (The Information Retrieval Experiment Platform) is an open-source ecosystem designed to address the reproducibility crisis and data leakage in Information Retrieval (IR) research. By executing dockerized retrieval models in sandboxed environments, TIREx facilitates truly "blind" experimentation, preventing Large Language Model (LLM) contamination and providing a standardized, "always-on" leaderboard. This paper details the platform's architecture, its integration of standard IR tools, and a large-scale evaluation involving 1.9 billion documents that demonstrates its scalability and the task-specific nature of IR model performance.

Introduction

Information Retrieval research relies heavily on the "Cranfield paradigm"—comparing retrieval approaches on standardized datasets. However, the field faces three major hurdles: the reproducibility crisis caused by environment dependencies, a lack of blinded experimentation leading to potential overfitting, and LLM contamination where publicly available test sets are leaked into training data. TIREx addresses these by providing a cloud-native execution framework that ensures dependencies are self-contained and test data remains hidden from the models during execution.

Methods and Infrastructure

TIREx solves these problems by integrating industry-standard IR tools—ir_datasets for data access, PyTerrier for pipeline construction, and ir_measures for evaluation—into a single framework called TIRA.

The TIRA process within the TIREx ecosystem, illustrating the interaction between organizers, participants, and researchers.

Researchers submit their models as Docker images, which TIREx runs in a sandbox with no internet access. This ensures all dependencies are self-contained and prevents data leakage. The platform treats retrieval as a sequence of modular steps (e.g., indexing, first-stage retrieval, re-ranking), allowing intermediate artifacts to be cached and reused, which significantly reduces computational costs.

Experimental Setup

The platform's capabilities were demonstrated through a massive evaluation involving 15 distinct datasets (including TREC Deep Learning, Touché, and NFCorpus) and 32 shared tasks. In total, the evaluation covered 1.9 billion documents and 50 "standard" retrieval approaches imported from frameworks like BEIR, Pyserini, and ChatNoir.

Key Results and Discussion

TIREx proved its scalability by executing 1,600 individual runs (50 approaches across 32 tasks) in less than a week on a midsize cluster. Beyond efficiency, the platform enables "always-on" benchmarking, allowing new models to be compared against historical baselines at any time.

Reproducibility analysis across various IR tasks, showing success rates and effect ratios for system preferences.

A key finding from the reproducibility analysis using the repro_eval tool showed that while system preferences (e.g., "Model A is better than Model B") were highly reproducible between similar tasks like TREC DL 2019 and 2020 (88.1% success rate), this rate dropped as low as 31% when moving to older or more distinct web search tasks. This underscores the necessity of the multi-task evaluation that TIREx enables, as "state-of-the-art" performance is often highly task-specific.

Conclusion and Future Work

TIREx provides a robust infrastructure for protecting the integrity of IR benchmarks against LLM contamination. Future developments will focus on integrating qualitative analysis tools like DiffIR to visualize document-level ranking differences and linking leaderboards directly to the IR Anthology to provide live, reproducible performance metrics for published research.

Summary updated

BibTeX
@inproceedings{frobe2024information,
  author = {Maik Fröbe and Jan Heinrich Reimer and Sean MacAvaney and Niklas Deckers and Simon Reich and Janek Bevendorff and Benno Stein and Matthias Hagen and Martin Potthast},
  title = {{The Information Retrieval Experiment Platform (Extended Abstract)}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
}
Figure