Overview of the TREC 2023 Deep Learning Track

Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Hossein A. Rahmani, Daniel Campos, Jimmy Lin, Ellen M. Voorhees, Ian Soboroff

Abstract

The fifth and final year of the TREC Deep Learning Track (2023) marks a significant milestone in the field of information retrieval. This paper summarizes the track's findings, most notably the official transition of Large Language Model (LLM) prompting approaches to the state-of-the-art for ranking tasks, surpassing previously dominant fine-tuned neural models. Furthermore, the track explored the efficacy of synthetic queries generated by models like GPT-4 and T5, concluding that these AI-generated queries can produce system rankings nearly identical to those derived from real human queries, offering a scalable path for future evaluation.

Introduction

Modern search engine evaluation faces a scalability crisis. As retrieval systems improve, they identify vast numbers of relevant documents within massive corpora, making it increasingly difficult and expensive for human judges to provide comprehensive relevance labels. This leads to "score saturation" and incomplete benchmarks. The TREC 2023 Deep Learning Track addressed these challenges by benchmarking the latest zero-shot and few-shot LLM prompting techniques against traditional and fine-tuned neural models, and by investigating the use of synthetic queries to build reusable test collections.

Methodology and Datasets

The track focused on two primary tasks: passage ranking and document ranking. Participants submitted runs for either end-to-end retrieval or top-100 reranking. To manage the human judging budget, organizers employed passage clustering to avoid redundant assessments.

A significant innovation this year was the introduction of synthetic queries. The process involved sampling passages, using GPT-4 to filter for standalone coherence, and then generating queries using both fine-tuned T5 and zero-shot GPT-4 prompts. These were evaluated alongside real human queries from Bing.

The track utilized the MS MARCO v2 dataset, featuring a corpus of 138 million passages and 11.9 million documents. Evaluation was conducted on a final set of 82 queries, comprising a mix of human and synthetic origins. Runs were categorized into four groups:

  • trad: Traditional methods (e.g., BM25).
  • nn: Neural networks without pre-training.
  • nnlm: Pre-trained fine-tuned models (e.g., BERT-style).
  • prompt: LLM prompting approaches.

Key Results: The LLM Takeover

The most striking result of the 2023 track was the clear dominance of LLM prompting. While fine-tuned neural models (nnlm) had led the field for the previous four years, prompt-based runs established a new performance ceiling.

Performance comparison (NDCG@10) for passage and document ranking tasks across different model categories.

As shown in the figure above, the best prompt run achieved an NDCG@10 of 0.6994 in passage ranking, significantly outperforming the best nnlm run (0.5972) and dwarfing traditional methods (0.2825). This trend was consistent across both passage and document ranking tasks.

Insights into Model Performance and Synthetic Evaluation

The shift toward prompting is not just visible in aggregate scores but also at the individual query level.

Per-query comparison of NDCG@10 between the best prompt-based run and the best fine-tuned neural model (nnlm) run.

The per-query analysis reveals that prompt models frequently outperform nnlm models across a wide variety of search intents. Beyond raw performance, the track provided critical evidence for the validity of synthetic queries. System rankings produced using synthetic queries correlated highly with those using real human queries (Kendall’s $\tau = 0.8487$). This suggests that AI-generated benchmarks are a viable, cost-effective alternative to human-labeled datasets.

Interestingly, the study found no "home field" advantage; systems did not perform better on queries generated by the same model architecture they utilized (e.g., T5-based rankers did not have an unfair advantage on T5-generated queries). However, GPT-4 queries were notably more verbose, averaging 10.72 words compared to the ~5.7 words of human or T5 queries.

Conclusion and Future Work

The TREC 2023 Deep Learning Track concludes with the clear message that prompting is the new standard for high-performance ranking. While synthetic queries show immense promise for scaling evaluation, they still require human oversight to filter noise—only about 27-37% of generated queries were deemed usable. Future research must continue to refine synthetic generation and monitor for subtle biases as the community moves toward AI-assisted benchmarking.

Summary updated

Abstract

This is the fifth year of the TREC Deep Learning track. As in previous years, we leverage the MS MARCO datasets that made hundreds of thousands of human-annotated training labels available for both passage and document ranking tasks. We mostly repeated last year's design, to get another matching test set, based on the larger, cleaner, less-biased v2 passage and document set, with passage ranking as primary and document ranking as a secondary task (using labels inferred from passage). As we did last year, we sample from MS MARCO queries that were completely held out, unused in corpus construction, unlike the test queries in the first three years. This approach yields a more difficult test with more headroom for improvement. Alongside the usual MS MARCO (human) queries from MS MARCO, this year we generated synthetic queries using a fine-tuned T5 model and using a GPT-4 prompt. The new headline result this year is that runs using Large Language Model (LLM) prompting in some way outperformed runs that use the "nnlm" approach, which was the best approach in the previous four years. Since this is the last year of the track, future iterations of prompt-based ranking can happen in other tracks. Human relevance assessments were applied to all query types, not just human MS MARCO queries. Evaluation using synthetic queries gave similar results to human queries, with system ordering agreement of $τ=0.8487$. However, human effort was needed to select a subset of the synthetic queries that were usable. We did not see clear evidence of bias, where runs using GPT-4 were favored when evaluated using synthetic GPT-4 queries, or where runs using T5 were favored when evaluated on synthetic T5 queries.
BibTeX
@inproceedings{craswell2023overview,
  author = {Nick Craswell and Bhaskar Mitra and Emine Yilmaz and Hossein A. Rahmani and Daniel Campos and Jimmy Lin and Ellen M. Voorhees and Ian Soboroff},
  title = {{Overview of the TREC 2023 Deep Learning Track}},
  booktitle = {TREC},
  year = {2023},
  url = {https://openreview.net/forum?id=wqZNakRUMy},
}
Figure