Overview of the TREC 2023 deep learning track

Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Hossein A. Rahmani, Daniel Campos, Jimmy Lin, Ellen M. Voorhees, Ian Soboroff

Abstract

The fifth and final year of the TREC Deep Learning track continued to leverage the MS MARCO datasets, specifically the larger and cleaner v2 passage and document sets. This year introduced a major shift in methodology: the use of Large Language Model (LLM) prompting, which officially outperformed the fine-tuned neural language model (nnlm) approaches that dominated previous years. Additionally, the track explored the use of synthetic queries generated by T5 and GPT-4 models. Results indicate that synthetic queries are highly effective for system evaluation, correlating strongly with human-judged performance ($\tau=0.8487$), although human oversight remains necessary to ensure query quality.

Introduction

Information Retrieval (IR) evaluation at scale requires vast amounts of labeled data. The TREC Deep Learning track addresses this "data hunger" by providing benchmarks based on the MS MARCO dataset. In 2023, the track utilized the MS MARCO v2 corpus, featuring 138 million passages and 11.9 million documents. A key goal was to move beyond "score saturation" by using harder, held-out queries and investigating synthetic query generation to augment human evaluation efforts.

Methodology and Participation

The track focused on two primary tasks: Passage Ranking and Document Ranking. Participants categorized their submissions based on the underlying technology:

  • trad: Traditional IR methods like BM25.
  • nn: Simple neural representation learning.
  • nnlm: Pre-trained models (e.g., BERT) used as cross-encoders and fine-tuned on ranking labels.
  • prompt: LLMs (e.g., GPT-4) used via prompting for re-ranking, rather than traditional fine-tuning.
Summary of run categories and participation for the 2023 track.

As shown in the participation summary, the "prompt" category saw significant adoption in the passage ranking task, while "nnlm" remained the most common approach for document ranking. Synthetic queries were generated using a fine-tuned T5 model and zero-shot GPT-4 prompts, then filtered by human NIST assessors to ensure usability.

Key Results: The Rise of Prompting

The most significant finding of the 2023 track was the superior performance of LLM prompting. In previous years, fine-tuned cross-encoders (nnlm) were the state-of-the-art. This year, the "prompt" runs achieved the highest scores in both tasks.

Comparison of NDCG@10 scores for passage and document ranking tasks, highlighting the performance gap between prompt-based and nnlm-based runs.

In the passage ranking task, the top "prompt" run achieved an NDCG@10 of 0.6994, significantly higher than the best "nnlm" run (0.5972). This trend was mirrored in the document ranking task, where the single "prompt" run outperformed all "nnlm" competitors. Furthermore, synthetic queries proved to be effective "hard" tests for retrieval systems, providing more headroom for performance measurement without introducing clear "self-preference" bias (e.g., GPT-4 based rankers favoring GPT-4 generated queries).

Discussion and Conclusion

The success of synthetic queries is a major milestone, suggesting that AI can assist in building its own evaluation benchmarks. However, the process is not yet fully automated; only about a third of synthetic queries were deemed usable by human assessors. Furthermore, while prompting-based rankers are highly effective, their high computational cost often limits them to a re-ranking role rather than full-corpus search.

As the TREC Deep Learning track concludes, it leaves a legacy of large-scale datasets and a clear signal that the field is moving toward LLM-driven reasoning for complex retrieval tasks. Future research will likely focus on improving the efficiency of these models and further refining synthetic evaluation pipelines.

Summary updated

Abstract

This is the fifth year of the TREC Deep Learning track. As in previous years, we leverage the MS MARCO datasets that made hundreds of thousands of human-annotated training labels available for both passage and document ranking tasks. We mostly repeated last year's design, to get another matching test set, based on the larger, cleaner, less-biased v2 passage and document set, with passage ranking as primary and document ranking as a secondary task (using labels inferred from passage). As we did last year, we sample from MS MARCO queries that were completely held out, unused in corpus construction, unlike the test queries in the first three years. This approach yields a more difficult test with more headroom for improvement. Alongside the usual MS MARCO (human) queries from MS MARCO, this year we generated synthetic queries using a fine-tuned T5 model and using a GPT-4 prompt. The new headline result this year is that runs using Large Language Model (LLM) prompting in some way outperformed runs that use the "nnlm" approach, which was the best approach in the previous four years. Since this is the last year of the track, future iterations of prompt-based ranking can happen in other tracks. Human relevance assessments were applied to all query types, not just human MS MARCO queries. Evaluation using synthetic queries gave similar results to human queries, with system ordering agreement of $τ=0.8487$. However, human effort was needed to select a subset of the synthetic queries that were usable. We did not see clear evidence of bias, where runs using GPT-4 were favored when evaluated using synthetic GPT-4 queries, or where runs using T5 were favored when evaluated on synthetic T5 queries.
BibTeX
@inproceedings{craswell2023overview,
  author = {Nick Craswell and Bhaskar Mitra and Emine Yilmaz and Hossein A. Rahmani and Daniel Campos and Jimmy Lin and Ellen M. Voorhees and Ian Soboroff},
  title = {{Overview of the TREC 2023 deep learning track}},
  booktitle = {TREC},
  year = {2023},
  url = {https://openreview.net/forum?id=wqZNakRUMy},
}
Figure