Overview of the TREC 2023 Tip-of-the-Tongue Track

Fernando Diaz, Jaime Arguello, Samarth Bhargav, Fernando Diaz 0001, Evangelos Kanoulas, Bhaskar MitraTrec 2023

Introduction

The "Tip-of-the-Tongue" (ToT) state is a common cognitive phenomenon where an individual can recall various attributes of an item—such as the plot of a movie or specific actors—but cannot retrieve its name. In the context of Information Retrieval (IR), these "known-item" searches present a unique challenge. Unlike standard keyword-based queries, ToT requests are typically verbose, noisy, and contain a mix of semantic facts, episodic context, and even false memories. The TREC 2023 Tip-of-the-Tongue track was established to benchmark the ability of modern IR systems to navigate these complex, human-centric information needs within the movie domain.

Methodology and Datasets

The track utilized the MS-ToT Dataset, which consists of query-answer pairs sourced from the "I Remember This Movie..." community Q&A site. The retrieval corpus comprised approximately 231,000 Wikipedia pages related to audiovisual works. Participants were provided with sentence-level annotations identifying plot descriptions, expressions of uncertainty, and exclusion criteria.

To establish performance benchmarks, three baselines were provided:

  1. BM25: A standard lexical matching algorithm.
  2. Dense Retrieval (DistilBERT): A neural approach matching semantic meanings.
  3. GPT-4: A generative baseline that predicted movie titles directly from queries, which were then mapped to the corpus.

Top-performing teams often employed multi-stage pipelines, utilizing initial retrieval followed by sophisticated re-ranking. Many also leveraged external knowledge bases (IMDb, Wikidata) and pre-trained models on specialized datasets like TOMT-KIS and Reddit-TOMT to better handle the nuances of ToT language.

Key Results and Insights

The evaluation focused on metrics such as NDCG@10, NDCG@1000, and MRR@1000. The results highlighted a significant performance gap between specialized retrieval systems and general baselines.

Performance comparison of various participant runs and baselines across NDCG@10, NDCG@1000, and MRR@1000.

The top-performing run, dpr-1000-rerank-robin by CMU-LTI, achieved an NDCG@1000 of 0.5554, significantly outperforming the next best group (0.3301). Interestingly, the GPT-4 baseline proved highly competitive in precision (NDCG@1000 of 0.2624), surpassing 26 out of 33 participant runs, though it lagged behind in recall (0.3733 vs. 0.7933 for the winner).

The Verbosity Paradox and False Memories

A critical finding was the negative correlation between query length and retrieval performance. While more information might intuitively seem helpful, longer queries often introduced "noise"—such as social niceties or false memories—that confused standard encoders. Qualitative analysis revealed that queries containing false memories (e.g., misremembering a character's nationality) frequently resulted in a mean score of zero, as current systems lack the capability to "fact-check" user input against the corpus.

Distribution of performance metrics across individual test queries, illustrating the high variance and difficulty of specific ToT requests.

The variance across queries (shown above) underscores the difficulty of the task; while some items are easily found, many queries remain completely unserved by current state-of-the-art models.

Discussion and Future Work

The TREC 2023 ToT track identified several key areas for future research:

  • Granularity: Improving retrieval when queries refer to specific TV episodes rather than entire series.
  • Handling False Memories: Developing models capable of identifying and discounting potentially incorrect information provided by the user.
  • Query Decomposition: Exploring methods to strip away conversational "noise" and isolate the core searchable kernels of information.

While modern systems are making strides in finding the "unforgettable," bridging the gap between messy human memory and structured digital corpora remains a formidable challenge for the IR community.

Summary updated

BibTeX
@inproceedings{diaz2023overview,
  author = {Fernando Diaz and Jaime Arguello and Samarth Bhargav and Fernando Diaz and Evangelos Kanoulas and Bhaskar Mitra},
  title = {{Overview of the TREC 2023 Tip-of-the-Tongue Track}},
  booktitle = {Trec},
  year = {2023},
}
Figure