CONVERSER: Few-shot Conversational Dense Retrieval with Synthetic Data Generation

Chao-Wei Huang, Chen-Yu Hsu, Tsu-Yuan Hsu, Chen-An Li, Yun-Nung Chen

TL;DR

This paper introduces CONVERSER, a novel framework that uses large language models (LLMs) to generate synthetic conversational data for training dense retrieval models. By leveraging LLMs with just a handful of examples, CONVERSER achieves performance comparable to fully-supervised models trained on thousands of dialogues, significantly reducing the data requirements for conversational information retrieval.

What Problem Does This Paper Address?

Conversational Information Retrieval (CIR) aims to provide users with relevant information through natural, multi-turn dialogues. Dense retrieval models have shown great promise in this area due to their ability to understand query semantics. However, training these models requires a substantial amount of in-domain conversational data, which is expensive and time-consuming to collect. This data scarcity is a major bottleneck for developing effective conversational dense retrievers, especially for new domains. Existing methods often still rely on large datasets for data augmentation, failing to fully address the core problem of limited in-domain data.

Methods

CONVERSER tackles the data scarcity problem by generating synthetic conversational query-passage pairs using LLMs. The core of the method is few-shot conversational query generation. It leverages the in-context learning capabilities of LLMs by providing them with a few example conversations (k examples) and a target passage. The LLM is then prompted to generate a series of queries that are relevant to the passage and continue the conversational flow.

To address the unique nature of conversational queries, CONVERSER employs a two-stage generation process:

  1. First Query Generation: A specific template is used to generate the initial, self-contained query based on the passage.
  2. Follow-up Query Generation: Once the first query is generated, subsequent queries are generated in a conversational manner, building upon the previous turns and the passage.

Additionally, the framework incorporates passage switching to simulate how relevant passages might change within a conversation, and a consistency filtering mechanism to ensure the generated queries are grounded and reliable by checking for round-trip consistency.

Datasets

The paper evaluates CONVERSER on two benchmark datasets:

  • OR-QuAC (Qu et al., 2020): A conversational question answering dataset.
  • TREC CAST 19 (Dalton et al., 2019): A conversational search benchmark.

The retrieval corpora used for generating synthetic data include:

  • 11 million passages from English Wikipedia for OR-QuAC.
  • 8 million webpage passages from MS-MARCO and 30 million Wikipedia passages from TREC-CAR for TREC CAST-19.

Key Insights and Ideas

  • LLMs as powerful synthetic data generators: The paper demonstrates that LLMs, even with minimal few-shot examples, can generate high-quality synthetic conversational data that is effective for training dense retrievers.
  • Two-stage generation is crucial: Separating the generation of the first query from follow-up queries significantly improves the quality and coherence of the synthetic conversations, ensuring the initial query is self-contained.
  • Few-shot learning for data augmentation: CONVERSER proves that effective data augmentation for dense retrieval doesn't require massive amounts of in-domain data; a small set of examples can suffice.
  • Adaptability across domains: The method shows promise for adapting to new conversational search tasks with minimal domain-specific data, a significant advantage over traditional transfer learning approaches.

Key Results

  • On the OR-QuAC dataset, CONVERSER, trained on synthetic data from just 6 examples, achieved performance comparable to supervised DPR models trained on 4000 annotated dialogues.
  • For TREC CAST-19, CONVERSER outperformed supervised DPR trained on OR-QuAC, highlighting its task-specific generation strategy's effectiveness.
  • An ablation study showed that all proposed components (two-stage generation, passage switching, consistency filtering) contribute to the overall performance improvement.
  • The performance of CONVERSER scales with the amount of generated data, indicating the crucial role of synthetic conversational data.

Limitations and Future Work

  • LLM choice and instruction-tuning: The authors note that while they used LLaMA-13B, instruction-tuned LLMs did not perform well for conversational query generation in their experiments. Future work could explore better instruction-following LLMs.
  • Filtering mechanisms: While consistency filtering is employed, the authors suggest exploring better filtering mechanisms to further improve the quality of generated data.
  • Specialized data generation: Future research could focus on synthesizing more specialized data for conversational dense retrieval, such as query rewrites.
  • Cross-validation for CAST-19: Due to resource constraints, the authors could not perform 5-fold cross-validation on CAST-19, using only 15 out of 20 conversations for evaluation.

Summary updated

Abstract

Conversational search provides a natural interface for information retrieval (IR). Recent approaches have demonstrated promising results in applying dense retrieval to conversational IR. However, training dense retrievers requires large amounts of in-domain paired data. This hinders the development of conversational dense retrievers, as abundant in-domain conversations are expensive to collect. In this paper, we propose CONVERSER, a framework for training conversational dense retrievers with at most 6 examples of in-domain dialogues. Specifically, we utilize the in-context learning capability of large language models to generate conversational queries given a passage in the retrieval corpus. Experimental results on conversational retrieval benchmarks OR-QuAC and TREC CAsT 19 show that the proposed CONVERSER achieves comparable performance to fully-supervised models, demonstrating the effectiveness of our proposed framework in few-shot conversational dense retrieval. All source code and generated datasets are available at https://github.com/MiuLab/CONVERSER
BibTeX
@inproceedings{huang2023converser,
  author = {Chao-Wei Huang and Chen-Yu Hsu and Tsu-Yuan Hsu and Chen-An Li and Yun-Nung Chen},
  title = {{CONVERSER: Few-shot Conversational Dense Retrieval with Synthetic Data Generation}},
  booktitle = {SIGDIAL},
  year = {2023},
  url = {https://openreview.net/forum?id=qx50rM3lbB},
}
Figure