Abstract
TIREx (The Information Retrieval Experiment Platform) is an open-source ecosystem designed to address the reproducibility crisis and data leakage in Information Retrieval (IR) research. By executing dockerized retrieval models in sandboxed environments, TIREx facilitates truly "blind" experimentation, preventing Large Language Model (LLM) contamination and providing a standardized, "always-on" leaderboard. This paper details the platform's architecture, its integration of standard IR tools, and a large-scale evaluation involving 1.9 billion documents that demonstrates its scalability and the task-specific nature of IR model performance.
Introduction
Information Retrieval research relies heavily on the "Cranfield paradigm"—comparing retrieval approaches on standardized datasets. However, the field faces three major hurdles: the reproducibility crisis caused by environment dependencies, a lack of blinded experimentation leading to potential overfitting, and LLM contamination where publicly available test sets are leaked into training data. TIREx addresses these by providing a cloud-native execution framework that ensures dependencies are self-contained and test data remains hidden from the models during execution.
Methods and Infrastructure
TIREx solves these problems by integrating industry-standard IR tools—ir_datasets for data access, PyTerrier for pipeline construction, and ir_measures for evaluation—into a single framework called TIRA.
Researchers submit their models as Docker images, which TIREx runs in a sandbox with no internet access. This ensures all dependencies are self-contained and prevents data leakage. The platform treats retrieval as a sequence of modular steps (e.g., indexing, first-stage retrieval, re-ranking), allowing intermediate artifacts to be cached and reused, which significantly reduces computational costs.
Experimental Setup
The platform's capabilities were demonstrated through a massive evaluation involving 15 distinct datasets (including TREC Deep Learning, Touché, and NFCorpus) and 32 shared tasks. In total, the evaluation covered 1.9 billion documents and 50 "standard" retrieval approaches imported from frameworks like BEIR, Pyserini, and ChatNoir.
Key Results and Discussion
TIREx proved its scalability by executing 1,600 individual runs (50 approaches across 32 tasks) in less than a week on a midsize cluster. Beyond efficiency, the platform enables "always-on" benchmarking, allowing new models to be compared against historical baselines at any time.
A key finding from the reproducibility analysis using the repro_eval tool showed that while system preferences (e.g., "Model A is better than Model B") were highly reproducible between similar tasks like TREC DL 2019 and 2020 (88.1% success rate), this rate dropped as low as 31% when moving to older or more distinct web search tasks. This underscores the necessity of the multi-task evaluation that TIREx enables, as "state-of-the-art" performance is often highly task-specific.
Conclusion and Future Work
TIREx provides a robust infrastructure for protecting the integrity of IR benchmarks against LLM contamination. Future developments will focus on integrating qualitative analysis tools like DiffIR to visualize document-level ranking differences and linking leaderboards directly to the IR Anthology to provide live, reproducible performance metrics for published research.
Summary updated