Joint Inference of Retrieval and Generation for Passage Re-ranking

Wei Fang, Yung-Sung Chuang, James R. GlassEACL (Findings) 2024 · OpenReview

Abstract

Passage retrieval is a crucial component of modern open-domain question answering (QA) systems, providing information for downstream QA components to generate accurate and transparent answers. In this study we focus on passage re-ranking, proposing a simple yet effective method, Joint Passage Re-ranking (JPR), that optimizes the mutual information between query and passage distributions, integrating both cross-encoders and generative models in the re-ranking process. Experimental results demonstrate that JPR outperforms conventional re-rankers and language model scorers in both open-domain QA retrieval settings and diverse retrieval benchmarks under zero-shot settings.

Introduction

In open-domain question answering (QA), systems must navigate massive databases to identify specific passages that contain the answer to a user's query. This process typically involves a two-stage pipeline: a high-recall retriever identifies a set of candidate passages, and a high-precision re-ranker sorts these candidates to find the most relevant text.

Traditional re-rankers, often implemented as discriminative cross-encoders, frequently suffer from "popularity bias." These models tend to assign high scores to "generic" passages—those containing broad summaries or common information—because such text appears statistically likely across many contexts. While these passages are high-probability, they often lack the specific details required to answer niche queries. Conversely, generative models that score passages based on the likelihood of generating the query often underperform on their own. To address this, we introduce Joint Passage Re-ranking (JPR), a method that fuses discriminative and generative approaches by optimizing for Pointwise Mutual Information (PMI).

Methodology

The core innovation of JPR is the transition from simple likelihood scoring to a Mutual Information (MI) framework.

The Objective

Instead of relying solely on the conditional probability of a passage given a query ($p(z|x)$), JPR maximizes the Pointwise Mutual Information (PMI). This metric measures the degree of association between a query and a passage relative to their independent occurrences, effectively penalizing "generic" passages that have a high unconditional probability ($p(z)$).

Joint Inference and Fine-tuning

Using Bayes' Theorem, the authors derive an inference formula that functions as a convex combination of two distinct scores:

  1. Discriminative Score: Provided by a BERT-based cross-encoder.
  2. Generative Score: Provided by a generative language model (e.g., T5 or LLaMA).

To further align these components, the authors propose Joint Fine-tuning (JPR-FT). This involves a joint training objective that incorporates a "matching loss" using symmetric KL divergence. This constraint ensures that the retrieval and generation distributions remain mathematically consistent during the training process.

Experimental Setup

JPR was evaluated across two distinct environments:

  • Supervised Open-Domain QA: Testing on the Natural Questions (NQ) and TriviaQA datasets against a corpus of 21 million Wikipedia passages.
  • Zero-Shot Retrieval: Testing on the BEIR benchmark, which includes 14 diverse datasets covering domains such as finance (FiQA), COVID-19 (TREC-COVID), and fact-checking (FEVER).

Results and Discussion

Performance in Supervised QA

JPR demonstrates superior performance compared to standard re-rankers and generative baselines. On the Natural Questions dataset, JPR-FT achieved a Top-1 accuracy of 51.4%, surpassing the standard BERT-FT re-ranker (49.4%).

Table 1: Performance comparison on Natural Questions and TriviaQA datasets. JPR and JPR-FT consistently outperform baselines in Top-1 accuracy.

Zero-Shot Robustness

The method's effectiveness extends to zero-shot scenarios. On the BEIR benchmark, JPR achieved an average nDCG@10 of 51.2, a significant improvement over the BERT-FT baseline of 49.4. JPR outperformed the baseline on 10 out of the 14 tasks in the suite.

Table 2: Zero-shot performance (nDCG@10) on the BEIR benchmark across various datasets.

Key Insights

  • Synergy Over Scale: A notable finding is that combining a smaller generative model (like T5-base) with a cross-encoder can outperform much larger standalone models like LLaMA-33B. This highlights the value of joint inference strategies.
  • The Genericness Penalty: By utilizing PMI, the model naturally discounts passages that are "too famous" or overly general, favoring those with unique relevance to the specific query.

Limitations

While JPR offers significant performance gains, it introduces higher computational costs due to the requirement for generative scoring (autoregressive decoding) for each candidate passage. Additionally, the performance advantage of JPR over standard re-rankers is most pronounced at Top-1 and tends to diminish as the evaluation window expands (e.g., Top-20). Future work may involve applying the PMI objective to the initial dense retrieval stage to improve the entire pipeline's efficiency.

Summary updated

Abstract

Passage retrieval is a crucial component of modern open-domain question answering (QA) systems, providing information for downstream QA components to generate accurate and transparent answers. In this study we focus on passage re-ranking, proposing a simple yet effective method, Joint Passage Re-ranking (JPR), that optimizes the mutual information between query and passage distributions, integrating both cross-encoders and generative models in the re-ranking process. Experimental results demonstrate that JPR outperforms conventional re-rankers and language model scorers in both open-domain QA retrieval settings and diverse retrieval benchmarks under zero-shot settings.
BibTeX
@inproceedings{fang2024joint,
  author = {Wei Fang and Yung-Sung Chuang and James R. Glass},
  title = {{Joint Inference of Retrieval and Generation for Passage Re-ranking}},
  booktitle = {EACL (Findings)},
  year = {2024},
  url = {https://openreview.net/forum?id=F4hiBuxpx8},
}
Figure