Decomposing Complex Queries for Tip-of-the-tongue Retrieval

Kevin Lin, Kyle Lo, Joseph Gonzalez, Dan Klein

Introduction

Information retrieval systems typically excel when users provide precise keywords or exact titles. However, human memory often fails in specific ways, leading to the "tip-of-the-tongue" (TOT) phenomenon. In these scenarios, a user might remember vague contextual details—such as a book's cover art, a general plot point, or the approximate time they read it—without recalling the title or author. These queries are characterized by high complexity and low lexical overlap with the target documents, making them difficult for standard search engines to resolve.

Comparison of TOT queries vs. standard retrieval datasets showing lower lexical overlap and higher query length.

Methodology: The MOREL Framework

To address the challenges of TOT retrieval, this research introduces MOREL, a framework that treats retrieval as a task of multi-step decomposition and expert ensembling. The system operates in three primary stages:

  1. Query Decomposition: Utilizing a Large Language Model (LLM) like GPT-3.5, the system breaks down a long, conversational query into discrete "clues." These clues categorize information into specific facets such as visual descriptions of the cover, estimated publication dates, and potential (often incorrect) titles.
  2. Specialized Retrieval Experts: Each extracted clue is routed to a specialized model. A Cover Retriever uses CLIP to match text descriptions against a database of book covers. A Date Retriever applies symbolic logic to filter results by publication year. A Title Retriever (using Contriever) focuses on the user's guesses, while a Base Retriever processes the entire query against book descriptions.
  3. Ensembling: The scores from these diverse experts are aggregated using a weighted sum to produce a final ranked list of candidates.
The MOREL framework architecture, illustrating the flow from query decomposition to specialized retrieval experts.

The WhatsThatBook Dataset

The researchers developed a new benchmark, WhatsThatBook, consisting of 14,441 real-world query-book pairs sourced from the Goodreads "What's That Book?" community. This dataset is unique because it contains "solved" queries where community members successfully identified books based on the posters' vague descriptions. Each entry includes the original complex query, the target book's metadata, and its cover image, providing a rich ground truth for evaluating TOT retrieval systems.

Key Insights and Results

The evaluation focused on Recall@K, measuring how frequently the correct item appeared in the top results. Standard text-only baselines like BM25 struggled, achieving only 8.3% Recall@5. In contrast, the MOREL framework achieved 28.4% Recall@5, representing a significant improvement over holistic dense retrieval models like Contriever (26.5%).

Key findings include:

  • Visual Memory Integration: Using CLIP to bridge textual descriptions and actual cover images proved highly effective, capturing books that text-only searches missed.
  • LLMs as Logic Translators: LLMs successfully translated relative timeframes (e.g., "read in high school in 2002") into absolute publication date filters, effectively pruning the search space.
  • Decomposition Advantage: Breaking queries into facets allowed the system to leverage the specific strengths of different models (e.g., semantic matching vs. image-text alignment).

Conclusion and Future Work

The MOREL framework demonstrates that decomposing complex, conversational queries into structured facets significantly enhances retrieval performance for "tip-of-the-tongue" searches. Future research may explore dynamic weighting of ensemble scores based on query characteristics and extending this methodology to other media domains such as film and music.

Summary updated

Abstract

When re-finding items, users who forget or are uncertain about identifying details often rely on creative strategies for expressing their information needs---complex queries that describe content elements (e.g., book characters or events), information beyond the document text (e.g., descriptions of book covers), or personal context (e.g., when they read a book). Standard retrieval models that rely on lexical or semantic overlap between query and document text are challenged in such retrieval settings, known as tip-of-the-tongue (TOT) retrieval. We introduce a simple but effective framework for handling such complex queries by decomposing the query with an LLM into individual clues routing those as subqueries to specialized retrievers, and ensembling the results. Our approach takes advantage of off-the-shelf retrievers (e.g., CLIP for retrieving images of book covers) or incorporate retriever-specific logic (e.g., date constraints). We show that our framework incorporating query decomposition into retrievers can improve gold book recall up to 6\% absolute gain for Recall@5 on a new collection of 14,441 real-world query-book pairs from an online community for resolving TOT inquiries.
BibTeX
@inproceedings{lin2023decomposing,
  author = {Kevin Lin and Kyle Lo and Joseph Gonzalez and Dan Klein},
  title = {{Decomposing Complex Queries for Tip-of-the-tongue Retrieval}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  doi = {10.18653/v1/2023.findings-emnlp.367},
}
Figure