Query Generation with External Knowledge for Dense Retrieval

Sukmin Cho, Soyeong Jeong, Wonsuk Yang, Jong ParkACL 2022

TL;DR

This paper introduces QGEK, a novel method for generating synthetic queries that incorporate external knowledge to improve dense retrieval models. By converting documents into triplet-based templates and enriching them with external information, QGEK creates more human-like queries that lead to better retrieval performance, especially in zero-shot settings.

What Problem Does This Paper Address?

Dense retrieval models, which represent queries and documents in an embedding space, have shown great promise for information retrieval. However, they require a massive amount of labeled query-document pairs for training, which are expensive and difficult to obtain. While generating synthetic queries from documents is a common workaround, these synthetic queries often lack the "hidden information" or external context present in human-generated queries, leading to suboptimal retrieval performance. This gap is particularly pronounced in zero-shot scenarios where the model encounters new domains.

Methods

The core of QGEK lies in its approach to query generation. Instead of directly generating queries from a document, QGEK first interprets the document and its associated external knowledge into a triplet-based template of the form <Subject, Relation, Object>. This template is then fed into a pre-trained language model (PLM), such as BART, which has been fine-tuned on a dataset of triplet-query pairs.

Here's a breakdown of the key steps:

  • Triplet Interpretation: Documents are broken down into subject, relation, and object components. For example, a document about zebra mussels might yield <"zebra mussels", "location", "Canada">.
  • External Knowledge Integration: Crucially, QGEK augments these triplets with external knowledge. If the document only mentions "Canada," QGEK might add "Canada is a country in North America" from external sources. This enriched information is then formatted into a single-text template.
  • PLM Fine-tuning: A PLM is fine-tuned using these triplet-based templates and corresponding queries. This teaches the PLM to generate queries that reflect the structured information and external context.
  • Dense Retriever Training: The generated synthetic queries, along with their corresponding documents, are then used to train a dense retriever model (specifically, DPR in this paper).

This method differs from prior work by explicitly incorporating external knowledge into the query generation process, aiming to bridge the gap between synthetic and human-labeled queries.

Datasets

  • Natural Questions (NQ): Used for in-domain experiments and as the source for document-query pairs for training the query generator.
  • 13 Representative IR Datasets: Used for out-domain (zero-shot) evaluation. These include datasets for argument retrieval, entity retrieval, question answering, fact checking, and more. (See Table 1 in the paper for a full list).

Key Insights and Ideas

  • External Knowledge is Key for Human-like Queries: The paper strongly suggests that human-labeled queries often contain implicit external knowledge that is not directly present in the document. QGEK's success hinges on its ability to inject this external context.
  • Triplet Templates Simplify Knowledge Integration: Converting information into a structured triplet format makes it easier for PLMs to process and utilize both document content and external knowledge.
  • Improved Zero-Shot Performance: QGEK shows promise in improving dense retrieval performance in out-of-domain settings, where traditional methods often struggle. This highlights the value of richer query generation for generalization.
  • Unique Words as a Proxy for "Hidden Information": The analysis of unique words in queries suggests that QGEK-generated queries tend to have a distribution of unique words more similar to human-labeled queries, indicating they are more likely to ask about information not explicitly stated in the document.

Key Results

  • In-Domain Performance: While human-labeled queries (Gold) still slightly outperform QGEK in the in-domain NQ setting, QGEK shows significant improvement over a baseline that does not use external knowledge.
  • Out-Domain Performance: QGEK demonstrates comparable performance to gold queries in out-domain settings, and in some cases, even outperforms them. Specifically, QGEK-trained retrievers showed better performance on 4 out of 13 out-domain datasets.
  • Qualitative Analysis: Human evaluation indicated that QGEK-generated queries exhibit higher grammaticality and difficulty compared to gold labels, suggesting they require more reasoning and hidden information.
  • Unique Word Distribution: QGEK-generated queries showed a distribution of unique words more aligned with human-labeled queries than other synthetic query generation methods.

Limitations and Future Work

  • MS MARCO Dataset: The authors acknowledge that their method was trained on a relatively small dataset (SimpleQuestions) and that using a larger, more diverse dataset like MS MARCO could further improve performance. They suggest that QGEK can be applied orthogonally to MS MARCO for future work.
  • Template Complexity: The current triplet-based template is relatively simple. Future work aims to propose templates that can accommodate more complex logical structures for generating a wider variety of high-quality queries.
  • Optimizing for IR Tasks: While QGEK generates queries closer to human-labeled ones, future research will focus on generating queries that are not only human-like but also optimized specifically for IR tasks.

Summary updated

BibTeX
@inproceedings{cho2022query,
  author = {Sukmin Cho and Soyeong Jeong and Wonsuk Yang and Jong Park},
  title = {{Query Generation with External Knowledge for Dense Retrieval}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
}
Figure