Abstract
The fifth and final year of the TREC Deep Learning track continued to leverage the MS MARCO datasets, specifically the larger and cleaner v2 passage and document sets. This year introduced a major shift in methodology: the use of Large Language Model (LLM) prompting, which officially outperformed the fine-tuned neural language model (nnlm) approaches that dominated previous years. Additionally, the track explored the use of synthetic queries generated by T5 and GPT-4 models. Results indicate that synthetic queries are highly effective for system evaluation, correlating strongly with human-judged performance ($\tau=0.8487$), although human oversight remains necessary to ensure query quality.
Introduction
Information Retrieval (IR) evaluation at scale requires vast amounts of labeled data. The TREC Deep Learning track addresses this "data hunger" by providing benchmarks based on the MS MARCO dataset. In 2023, the track utilized the MS MARCO v2 corpus, featuring 138 million passages and 11.9 million documents. A key goal was to move beyond "score saturation" by using harder, held-out queries and investigating synthetic query generation to augment human evaluation efforts.
Methodology and Participation
The track focused on two primary tasks: Passage Ranking and Document Ranking. Participants categorized their submissions based on the underlying technology:
- trad: Traditional IR methods like BM25.
- nn: Simple neural representation learning.
- nnlm: Pre-trained models (e.g., BERT) used as cross-encoders and fine-tuned on ranking labels.
- prompt: LLMs (e.g., GPT-4) used via prompting for re-ranking, rather than traditional fine-tuning.
As shown in the participation summary, the "prompt" category saw significant adoption in the passage ranking task, while "nnlm" remained the most common approach for document ranking. Synthetic queries were generated using a fine-tuned T5 model and zero-shot GPT-4 prompts, then filtered by human NIST assessors to ensure usability.
Key Results: The Rise of Prompting
The most significant finding of the 2023 track was the superior performance of LLM prompting. In previous years, fine-tuned cross-encoders (nnlm) were the state-of-the-art. This year, the "prompt" runs achieved the highest scores in both tasks.
In the passage ranking task, the top "prompt" run achieved an NDCG@10 of 0.6994, significantly higher than the best "nnlm" run (0.5972). This trend was mirrored in the document ranking task, where the single "prompt" run outperformed all "nnlm" competitors. Furthermore, synthetic queries proved to be effective "hard" tests for retrieval systems, providing more headroom for performance measurement without introducing clear "self-preference" bias (e.g., GPT-4 based rankers favoring GPT-4 generated queries).
Discussion and Conclusion
The success of synthetic queries is a major milestone, suggesting that AI can assist in building its own evaluation benchmarks. However, the process is not yet fully automated; only about a third of synthetic queries were deemed usable by human assessors. Furthermore, while prompting-based rankers are highly effective, their high computational cost often limits them to a re-ranking role rather than full-corpus search.
As the TREC Deep Learning track concludes, it leaves a legacy of large-scale datasets and a clear signal that the field is moving toward LLM-driven reasoning for complex retrieval tasks. Future research will likely focus on improving the efficiency of these models and further refining synthetic evaluation pipelines.
Summary updated