Introduction
The 2019 Open-Source IR Replicability Challenge (OSIRRC) was established to address the persistent "replicability crisis" in Information Retrieval (IR). Researchers frequently compare novel algorithms against weak or poorly tuned baselines because state-of-the-art systems are often difficult to install, configure, or execute. OSIRRC 2019 introduced a technical framework called "the jig" to standardize the packaging and execution of search engines using Docker containerization, aiming to make "different team, same experimental setup" replicability a reality.
The "Jig" Framework
The core contribution of OSIRRC is "the jig," a Python-based orchestrator that manages the lifecycle of IR experiments by communicating with the Docker Engine. By defining a common Docker specification, the framework ensures that diverse search engines—regardless of whether they are written in Java, C++, or Python—can be evaluated under a unified interface.
The framework relies on standardized "hooks"—scripts at fixed locations within the Docker image that the jig triggers at specific stages:
- init: For downloading artifacts or compiling source code.
- index: For processing a document collection into a searchable index.
- search: For running a batch of queries and producing standard TREC-format results.
To handle licensing restrictions for datasets like TREC, the jig mounts external read-only directories into the container. Efficiency is maintained through Docker snapshots; once the index hook completes, the jig takes a snapshot, allowing researchers to run multiple search experiments with different parameters without re-indexing the entire collection.
Datasets and Evaluation
The challenge focused on established newswire collections to provide a rich context for comparison:
- TREC 2004 Robust Track: The primary target, chosen for its 15-year history of comparative literature.
- TREC 2017 & 2018 Common Core Tracks: Utilizing more recent corpora like the New York Times Annotated Corpus and the Washington Post Corpus.
Key Results
The workshop successfully produced a curated library of 17 Docker images from 13 international teams. This library covers a wide spectrum of IR technology, from traditional "bag of words" models to modern neural rankers and high-efficiency engines.
On the Robust 04 track, the challenge generated 49 replicable runs. As shown in the results, top-performing models included the BERT-based Birch and Anserini (utilizing BM25 with pseudo-relevance feedback). The availability of these images provides the community with a "living" set of high-quality, push-button baselines that can be easily integrated into future research workflows.
Discussion and Future Work
While OSIRRC 2019 demonstrated the feasibility of containerized IR replicability, several hurdles remain. The 2019 version of the jig lacked native GPU support, which is a significant bottleneck for modern deep learning-based models. Additionally, the train hook for hyperparameter tuning requires further maturity. The authors emphasize that technical tools are only half the battle; the IR community must adopt a cultural norm of associating papers with standardized Docker images to truly resolve the replicability crisis.
Summary updated