Overview of the TREC 2024 Tip-of-the-Tongue track

Fernando Diaz, Jaime Arguello, Samarth Bhargav, Fernando Diaz 0001, To Eun Kim, Yifan He, Evangelos Kanoulas, Bhaskar MitraTrec 2024

Introduction

The TREC 2024 Tip-of-the-Tongue (ToT) track addresses the "I know it when I see it" challenge in information retrieval—finding a specific entity when its name is forgotten. Unlike standard keyword searches, ToT queries are verbose and "fuzzy," incorporating semantic facts, episodic context (e.g., "I saw this on a plane in the 90s"), and human linguistic quirks like uncertainty or false memories. As users increasingly turn to community Q&A platforms like Reddit's r/tipofmytongue, this track aims to develop IR systems capable of handling such conversational complexity.

Methodology and Datasets

The task was framed as an ad-hoc retrieval challenge: participants generated ranked lists of 1,000 document IDs from a Wikipedia corpus of over 3.1 million articles covering movies, celebrities, and landmarks.

A key innovation this year was the use of Large Language Models (LLMs) for Synthetic Query Generation. To supplement real-world data, organizers prompted LLMs to role-play individuals with fuzzy memories, intentionally introducing casual tones, missing identifiers, and even incorrect details to mimic human cognitive errors.

Evaluation relied on NDCG@1000, MRR, and Recall. The dataset included:

  • MS-ToT: 150 real-world movie queries from community sites.
  • Synthetic Queries: 450 LLM-generated queries across movies, celebrities, and landmarks.
  • Training Data: 450 movie queries from previous TREC years.

Results and Analysis

The track evaluated 18 runs from 6 research groups. The top-performing run, rg4o_t100_test by group h2oloo, achieved an NDCG@1000 of 0.8159 and a Recall@1000 of 0.9250. This significantly outperformed the official baseline-dense model (NDCG@1000 of 0.2377), demonstrating the efficacy of advanced re-ranking and the strategic use of provided baselines.

Summary of run performance metrics and score distributions across queries.

Several critical insights emerged from the competition:

  • The Synthetic Gap: Systems performed notably better on synthetic queries than on real human queries. This suggests that while LLMs can mimic the style of a ToT request, they do not yet fully capture the idiosyncratic "messiness" of genuine human memory.
  • Synthetic Data Utility: Despite the performance gap, results on synthetic queries correlated well with real-world performance, validating synthetic data as a useful proxy for model development.
  • Transparency and Data Contamination: A significant challenge noted by organizers was the difficulty in ensuring test sets remained "clean," as participants could not guarantee their models hadn't encountered the test data during massive-scale LLM pre-training.

Conclusion and Future Work

While modern dense retrieval and re-ranking systems are making strides, real-world human memory remains a difficult target. Future iterations of the ToT track may focus on more adversarial synthetic generation to better model human confusion and expand the corpus beyond Wikipedia to include more obscure items. Additionally, addressing "unanswerable" queries—where the target item is absent from the database—remains a key frontier for the field.

Summary updated

BibTeX
@inproceedings{diaz2024overview,
  author = {Fernando Diaz and Jaime Arguello and Samarth Bhargav and Fernando Diaz and To Eun Kim and Yifan He and Evangelos Kanoulas and Bhaskar Mitra},
  title = {{Overview of the TREC 2024 Tip-of-the-Tongue track}},
  booktitle = {Trec},
  year = {2024},
}
Figure