Introduction
Tip-of-the-Tongue (ToT) queries represent a unique challenge in information retrieval. These queries are typically long, conversational, and characterized by "noise"—irrelevant details or uncertain hedging (e.g., "I think it was a car, or maybe a truck"). Standard search engines often struggle with these descriptions because they lack the exact keyword matches found in traditional search. The TREC 2023 Tip-of-the-Tongue Track task required identifying the correct Wikipedia article for a movie based on these vague, natural language descriptions.
Methodology
Researchers from CMU and the University of Lisbon developed a two-stage retrieval pipeline consisting of first-stage retrieval followed by large language model (LLM) re-ranking.
First-Stage Retrieval
The team utilized Dense Passage Retrieval (DPR), which represents queries and documents as high-dimensional vectors. Two variations were tested:
- DPR Max-P: A standard approach that segments documents into passages and uses the maximum passage score to represent the document.
- Abstract Retriever: This approach used GPT-3.5 to annotate query sentences, attempting to match "high-signal" sentences (like plot descriptions) specifically against the abstract sections of Wikipedia articles.
Second-Stage Re-ranking
To refine the initial results, GPT-4 was employed as a zero-shot re-ranker. To overcome context window limitations when processing a large pool of candidates, the team introduced a "Round-Robin" strategy. For the top 1,000 candidates, titles were split into 10 groups of 100 using round-robin assignment based on initial rank. This ensured each group contained a mix of high and low-ranked candidates. GPT-4 ranked each group, and the top 10 from each were then pooled for a final "championship" re-ranking of the top 100 titles.
Data Augmentation
Due to the small size of the TREC training set (150 queries), the authors curated an auxiliary dataset from the Reddit r/tipofmytongue subreddit. By extracting movie titles from 1.3 million "solved" posts using GPT-3.5 and mapping them to Wikipedia IDs, they created a high-quality training set of 83,914 movie queries.
Results and Analysis
The combination of DPR Max-P and the GPT-4 Round-Robin re-ranking strategy yielded the best performance. As shown in the results table, the addition of LLM re-ranking significantly boosted metrics across the board.
The "Abstract Retriever" performed significantly worse than the simpler DPR Max-P approach. This suggests that in ToT queries, even seemingly "noisy" parts of the description may contain subtle cues that dense retrievers can leverage, which are lost when restricting the search to abstracts.
When compared against other participants in the TREC 2023 track, the CMU-LTI approach demonstrated superior performance, outperforming the median results in the majority of test topics.
Conclusion and Future Work
The study reinforces the power of LLMs as zero-shot rankers, even when provided only with movie titles rather than full document text. The round-robin strategy proved effective for handling deep candidate pools within context limits. However, the high cost and latency of GPT-4 API calls remain a barrier for real-time production. Future research may focus on utilizing smaller, fine-tuned local LLMs and exploring more sophisticated ways to integrate different document sections beyond the abstract.
Summary updated