Relevance-assisted Generation for Robust Zero-shot Retrieval

Jihyuk Kim, Minsoo Kim, Joonsuk Park, Seung-won HwangEMNLP 2023 · DOI

TL;DR

This paper tackles the challenge of zero-shot retrieval, where models trained on one domain struggle to perform on unseen domains. They propose "Relevance-assisted Multi-query Domain Adaptation" (RaMDA), a method that generates more relevant pseudo-queries by first creating relevance explanations and then using these to guide query generation. This approach significantly improves robustness to domain shifts, outperforming existing methods on challenging benchmarks.

What Problem Does This Paper Address?

Dense retrieval models, which have shown impressive performance on in-domain tasks, often falter when applied to out-of-domain (OOD) datasets. This "out-of-domain generalization" gap is a major weakness. To bridge this gap, researchers have explored domain adaptation techniques, particularly pseudo-query generation (PQG). PQG involves creating synthetic queries from a target corpus to fine-tune retrieval models. However, the authors identify a critical flaw in existing PQG methods: the generated pseudo-queries are often irrelevant to their corresponding documents, and they can introduce biases that negatively impact generalization. This leads to insufficient document representation and poor performance on OOD tasks.

Methods

The paper introduces Relevance-assisted Multi-query Domain Adaptation (RaMDA), which addresses the limitations of standard PQG in two main ways:

  1. Relevance Explanation and Guided Generation: Instead of directly generating pseudo-queries, RaMDA first decomposes the process.
  2. Relevance Explanation ($Z_d$): It generates a set of terms that explain why a document is relevant to potential queries. This is achieved using models like SPLADE, which provide weighted terms.
  3. Relevance-Guided Generation: These relevance explanations ($Z_d$) are then used to guide the generation of multiple pseudo-queries ($Q$). This ensures the generated queries are more aligned with the document's content and topic.
  4. Multi-Query Document Representation: To overcome the bottleneck of a single vector representation for a document, RaMDA augments the document's vector with representations derived from the generated pseudo-queries. This multi-vector approach allows documents to be matched against a more diverse set of relevant queries at inference time.

Datasets

The experiments were conducted on the BEIR benchmark, a collection of diverse information retrieval tasks designed to evaluate zero-shot generalization. Specifically, the authors focused on datasets exhibiting significant distribution shifts from MS MARCO, which they termed "BEIR-BiasShift". These include NFCorpus, SCIDOCS, TREC-COVID, and Climate-Fever. For training the pseudo-query generator, they used MS MARCO.

Key Insights and Ideas

  • Pseudo-queries can hurt generalization: The core insight is that poorly generated pseudo-queries, far from helping, can actively harm a retriever's ability to generalize to new domains. This is a crucial counter-intuitive finding.
  • Biases in PQG are the culprit: The paper identifies two key biases in standard PQG:
  • Term Frequency Bias: PQG tends to favor terms frequent in the training data, missing rare but important domain-specific terms.
  • Diversity Bias: PQG can generate a homogeneous set of queries, failing to capture the diverse topics within a document, especially in domains with longer documents.
  • Decomposition is key: Breaking down pseudo-query generation into relevance explanation and then guided generation is more effective than a single-step process.
  • Multi-query representation matters: Augmenting document representations with multiple pseudo-query vectors improves their ability to capture diverse aspects of a document's relevance.
  • Preemptive filtering is superior: The proposed method's "preemptive filtering" (guiding generation) is significantly better than "post-filtering" (generating many queries and then selecting the best ones), which can even worsen performance.

Key Results

  • RaMDA achieved significant improvements on the BEIR-BiasShift datasets, outperforming existing domain-adaptive retrievers.
  • Compared to the baseline PQG approach (GenQ), RaMDA increased nDCG@10 by 2.4 points and Recall@100 by 4.6 points on average across the BEIR-BiasShift datasets.
  • The proposed method demonstrated robustness to distribution shifts, a key goal for OOD generalization.
  • Ablation studies showed that addressing both term frequency and diversity biases contributed significantly to performance gains. Removing randomly sampled queries led to the least degradation, highlighting the importance of targeted bias mitigation.
  • RaMDA achieved better efficiency-effectiveness trade-offs compared to GenQ, requiring fewer pseudo-query samples to outperform even domain-invariant baselines like COCO-DR.

Limitations and Future Work

  • Dataset bias analysis: While the paper analyzes biases on specific datasets, a more comprehensive analysis across a wider range of OOD datasets could be beneficial.
  • Pseudo-query generator: The paper uses T5 as the backbone for pseudo-query generation. Exploring other advanced generative models could potentially yield further improvements.
  • Computational cost: While RaMDA shows efficiency gains, the relevance explanation step adds some computational overhead. Future work could focus on optimizing this step.
  • Generalizability of RaMDA: While tested on BEIR, further evaluation on other OOD retrieval scenarios and benchmarks would be valuable.

Summary updated

BibTeX
@inproceedings{kim2023relevance,
  author = {Jihyuk Kim and Minsoo Kim and Joonsuk Park and Seung-won Hwang},
  title = {{Relevance-assisted Generation for Robust Zero-shot Retrieval}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  doi = {10.18653/v1/2023.emnlp-industry.67},
}
Figure