Abstract
The fifth and final year of the TREC Deep Learning Track (2023) marks a significant milestone in the field of information retrieval. This paper summarizes the track's findings, most notably the official transition of Large Language Model (LLM) prompting approaches to the state-of-the-art for ranking tasks, surpassing previously dominant fine-tuned neural models. Furthermore, the track explored the efficacy of synthetic queries generated by models like GPT-4 and T5, concluding that these AI-generated queries can produce system rankings nearly identical to those derived from real human queries, offering a scalable path for future evaluation.
Introduction
Modern search engine evaluation faces a scalability crisis. As retrieval systems improve, they identify vast numbers of relevant documents within massive corpora, making it increasingly difficult and expensive for human judges to provide comprehensive relevance labels. This leads to "score saturation" and incomplete benchmarks. The TREC 2023 Deep Learning Track addressed these challenges by benchmarking the latest zero-shot and few-shot LLM prompting techniques against traditional and fine-tuned neural models, and by investigating the use of synthetic queries to build reusable test collections.
Methodology and Datasets
The track focused on two primary tasks: passage ranking and document ranking. Participants submitted runs for either end-to-end retrieval or top-100 reranking. To manage the human judging budget, organizers employed passage clustering to avoid redundant assessments.
A significant innovation this year was the introduction of synthetic queries. The process involved sampling passages, using GPT-4 to filter for standalone coherence, and then generating queries using both fine-tuned T5 and zero-shot GPT-4 prompts. These were evaluated alongside real human queries from Bing.
The track utilized the MS MARCO v2 dataset, featuring a corpus of 138 million passages and 11.9 million documents. Evaluation was conducted on a final set of 82 queries, comprising a mix of human and synthetic origins. Runs were categorized into four groups:
- trad: Traditional methods (e.g., BM25).
- nn: Neural networks without pre-training.
- nnlm: Pre-trained fine-tuned models (e.g., BERT-style).
- prompt: LLM prompting approaches.
Key Results: The LLM Takeover
The most striking result of the 2023 track was the clear dominance of LLM prompting. While fine-tuned neural models (nnlm) had led the field for the previous four years, prompt-based runs established a new performance ceiling.
As shown in the figure above, the best prompt run achieved an NDCG@10 of 0.6994 in passage ranking, significantly outperforming the best nnlm run (0.5972) and dwarfing traditional methods (0.2825). This trend was consistent across both passage and document ranking tasks.
Insights into Model Performance and Synthetic Evaluation
The shift toward prompting is not just visible in aggregate scores but also at the individual query level.
The per-query analysis reveals that prompt models frequently outperform nnlm models across a wide variety of search intents. Beyond raw performance, the track provided critical evidence for the validity of synthetic queries. System rankings produced using synthetic queries correlated highly with those using real human queries (Kendall’s $\tau = 0.8487$). This suggests that AI-generated benchmarks are a viable, cost-effective alternative to human-labeled datasets.
Interestingly, the study found no "home field" advantage; systems did not perform better on queries generated by the same model architecture they utilized (e.g., T5-based rankers did not have an unfair advantage on T5-generated queries). However, GPT-4 queries were notably more verbose, averaging 10.72 words compared to the ~5.7 words of human or T5 queries.
Conclusion and Future Work
The TREC 2023 Deep Learning Track concludes with the clear message that prompting is the new standard for high-performance ranking. While synthetic queries show immense promise for scaling evaluation, they still require human oversight to filter noise—only about 27-37% of generated queries were deemed usable. Future research must continue to refine synthetic generation and monitor for subtle biases as the community moves toward AI-assisted benchmarking.
Summary updated