SNU LDILAB @ TREC Tip of the tongue 2023

Jongho Kim, Soona Hong, Seung-won HwangTREC 2023

Abstract

The SNU LDILAB team addressed the 2023 TREC "Tip-of-the-Tongue" (ToT) retrieval challenge, which involves identifying specific items from vague, noisy, or underspecified descriptions. By combining self-supervised learning derived from Wikipedia structure with Virtual Adversarial Training (VAT), the team developed a retriever that outperforms large language models like GPT-4. Their approach effectively bridges the gap between sparse user queries and detailed document content while maintaining robustness against irrelevant information.

Introduction

The "tip-of-the-tongue" phenomenon—recalling specific details of an item without its name—presents a significant challenge for standard Information Retrieval (IR) systems. These systems typically struggle with two primary issues: underspecification, where user queries lack the detail found in target documents, and irrelevant noise, where queries include conversational filler or non-textual descriptions (e.g., visual quirks) that do not appear in text-based databases. This paper details a two-stage strategy to navigate these hurdles using movie identification as a primary benchmark.

Methodology

1. Addressing Underspecification via Self-Supervision

To train a robust model without extensive human-labeled data, the authors leveraged the inherent structure of Wikipedia. They treated the abstract of a movie's page as the ground-truth document and utilized other sections (e.g., Plot, Settings) as "query surrogates." This process generated approximately 147,000 query-document pairs. Furthermore, the team employed document expansion, appending these surrogates to the original documents to enrich their representations and make them more discoverable via vague queries.

2. Robustness through Virtual Adversarial Training (VAT)

To handle irrelevant "noise," the team implemented a two-phase augmentation strategy. First, they used sentence-level annotations to crop irrelevant parts of queries during training. Second, they applied Virtual Adversarial Training (VAT). Instead of random noise, VAT introduces calculated adversarial perturbations to the model's embeddings. By forcing the model to produce consistent outputs for both original and "jittered" inputs using Kullback-Leibler (KL) divergence, the model learns to ignore semantic variations that do not alter core meaning.

Comparison of the proposed method against other self-supervised baselines like ICT and GenQ.

Key Results and Evaluation

The proposed model demonstrated significant performance gains over standard Dense Passage Retrieval (DPR) baselines and other self-supervised methods. As shown in the comparison with methods like Inverse Cloze Task (ICT) and GenQ, the section-based extraction method used by the SNU LDILAB team is better suited for the ToT domain.

An ablation study highlights the necessity of addressing both underspecification and noise. Starting from a baseline DPR nDCG@10 of 0.1433, the integration of self-supervised surrogates and VAT led to a final nDCG@10 of 0.3052 on the development set.

Ablation study showing performance improvements from the DPR baseline to the final model incorporating underspecification and irrelevance handling.

Discussion and Conclusion

The success of this approach underscores the utility of Wikipedia as a natural proxy for ToT problems. The team also demonstrated that VAT provides a more substantial performance boost than simply increasing batch sizes, which is particularly relevant for resource-constrained environments.

However, limitations remain. The method's reliance on structured Wikipedia data may limit its generalizability to less organized domains, such as legal or technical documentation. Additionally, the model struggled with "BM25 hard negatives," suggesting that further research is needed to help dense retrievers distinguish between highly similar but incorrect items in vector space. Future work will focus on these domain-transfer challenges and optimizing the computational overhead associated with document expansion.

Summary updated

BibTeX
@inproceedings{kim2023snu,
  author = {Jongho Kim and Soona Hong and Seung-won Hwang},
  title = {{SNU LDILAB @ TREC Tip of the tongue 2023}},
  booktitle = {TREC},
  year = {2023},
}
Figure