Yale NLP at TREC 2024: Tip-of-the-Tongue Track

Rohan Phanse, Gabrielle Kaili-May Liu, Arman CohanTREC 2024

Introduction

The "Tip-of-the-Tongue" (ToT) phenomenon describes the frustrating experience of being able to describe an entity—such as a movie, celebrity, or landmark—in detail without being able to recall its name. In Information Retrieval (IR), ToT queries present a unique challenge: they are typically verbose, conversational, and often contain "false memories" or subjective descriptions rather than precise keywords.

Researchers from Yale NLP addressed the TREC 2024 ToT track by developing a robust retrieval and reranking system designed to handle multi-domain queries. While movie-related ToT data is relatively accessible via platforms like Reddit, data for other domains like celebrities and landmarks is scarce. To overcome this, the team leveraged large-scale synthetic data generation alongside a sophisticated two-stage neural pipeline.

System Architecture

The proposed system utilizes a two-stage pipeline consisting of initial retrieval followed by a multi-step reranking process.

1. Retrieval (DPR)

The first stage employs Dense Passage Retrieval (DPR) to filter millions of potential entities down to the top 1,000 candidates. The team experimented with two architectural variations:

  • Generalist Model: A single DPR model trained across all domains simultaneously.
  • Routing Approach: A domain classifier (powered by GPT-4o mini) that routes queries to domain-specific "expert" DPR models.

Contrary to expectations, the generalist model outperformed the specialized experts, likely due to the increased data diversity during training which improved overall robustness.

2. Hybrid Reranking

To refine the initial results, a two-phase reranking strategy was implemented using GPT-4o mini:

  • Pointwise Reranking: Candidates are scored in batches of 20 on a scale of 1–10. This stage acts as a high-recall filter, identifying the top 100 most promising candidates.
  • Listwise Reranking: The top 100 candidates undergo a "round-robin" listwise comparison to determine the final ranked order. This hybrid approach significantly improved performance, raising the NDCG@1000 from 0.54 to 0.60.

Data Augmentation and Synthetic Generation

To address the lack of training data for non-movie domains, the team generated over 100,000 synthetic queries using GPT-4o mini.

Summary of the query datasets used for training and evaluation, highlighting the significant role of synthetic data in the Celebrity and Landmark domains.

Using Wikipedia as a source for "answers" (36k celebrities and 18k landmarks), the team prompted the LLM to generate queries across varying lengths and difficulty levels. Crucially, the prompts instructed the model to include "unhelpful vague details and false memories" for difficult queries to better simulate human cognitive lapses.

Examples of synthetic queries generated across different length and difficulty settings, demonstrating the inclusion of vague descriptions and conversational filler.

Experimental Results

The system was evaluated on the official TREC 2024 ToT test set. The best-performing configuration, dpr-pnt-lst-rerank, combined the generalist DPR model with the hybrid reranking pipeline.

Metric Score
NDCG@10 0.5708
NDCG@1000 0.6049
MRR@1000 0.5482
Recall@1000 0.8417

The results indicate that while DPR is effective at capturing the correct answer within the top 1,000 results (84% recall), the LLM-based reranking is essential for surfacing the correct entity to the top of the list.

Discussion and Future Work

The success of the "generalist" model suggests that multi-domain training provides a regularization effect that specialized models lack. Furthermore, the effectiveness of pointwise reranking as a precursor to listwise reranking highlights the utility of LLMs as efficient filters in large-scale retrieval tasks.

Future work will focus on scaling this approach to a broader range of domains and optimizing the cost-performance trade-off of LLM reranking. While GPT-4o mini is efficient, the computational overhead of reranking 1,000 candidates remains a bottleneck for real-time applications.

Summary updated

BibTeX
@inproceedings{phanse2024yale,
  author = {Rohan Phanse and Gabrielle Kaili-May Liu and Arman Cohan},
  title = {{Yale NLP at TREC 2024: Tip-of-the-Tongue Track}},
  booktitle = {TREC},
  year = {2024},
}
Figure