Abstract
The bi-encoder design of dense passage retriever (DPR) is a key factor to its success in open-domain question answering (QA), yet it is unclear how DPR's question encoder and passage encoder individually contributes to overall performance, which we refer to as the encoder attribution problem. The problem is important as it helps us identify the factors that affect individual encoders to further improve overall performance. In this paper, we formulate our analysis under a probabilistic framework called encoder marginalization, where we quantify the contribution of a single encoder by marginalizing other variables. First, we find that the passage encoder contributes more than the question encoder to in-domain retrieval accuracy. Second, we demonstrate how to find the affecting factors for each encoder, where we train DPR with different amounts of data and use encoder marginalization to analyze the results. We find that positive passage overlap and corpus coverage of training data have big impacts on the passage encoder, while the question encoder is mainly affected by training sample complexity under this setting. Based on this framework, we can devise data-efficient training regimes: for example, we manage to train a passage encoder on SQuAD using 60% less training data without loss of accuracy.
Introduction
In Open-Domain Question Answering (QA), the Dense Passage Retriever (DPR) has become a standard architecture. It utilizes a "bi-encoder" setup where one neural network encodes a user's question and another encodes millions of potential answer passages into a shared vector space. Despite its success, the dual-encoder system has largely been treated as a "black box." When a model underperforms, it is difficult to diagnose whether the failure lies with the question encoder (Q-encoder) or the passage encoder (P-encoder). This study introduces encoder marginalization to solve this attribution problem, allowing researchers to isolate and quantify the individual contributions of each component.
The Encoder Marginalization Framework
To isolate the contribution of a specific encoder, the authors propose a probabilistic framework. To evaluate a single Q-encoder, it is paired with a diverse set of P-encoders trained on different datasets. By averaging the retrieval scores across these varied pairings, the influence of any single P-encoder is "marginalized out," revealing the inherent quality of the Q-encoder.
Since training an infinite number of encoders is computationally impossible, the authors use a Monte-Carlo approximation. They approximate the "average" performance by using a representative sample of encoders trained on five distinct QA domains: Natural Questions (NQ), TriviaQA, WebQuestions (WQ), CuratedTREC, and SQuAD-1.1.
Attribution Analysis: The Passage Encoder as the Primary Driver
A key finding of this research is that the passage encoder is the primary driver of retrieval performance. Across all benchmark datasets, the P-encoder contributes significantly more to marginalized retrieval accuracy than the Q-encoder. This suggests that the "heavy lifting" of organizing the vector space and mapping document semantics falls primarily on the passage encoder.
As shown in Figure 2, replacing a P-encoder typically leads to a much larger drop in accuracy than replacing a Q-encoder. This insight shifts the focus for model improvement toward optimizing how we train passage representations.
Data Efficiency and the SQuAD Paradox
The study also explored how training volume affects each encoder. While the Q-encoder generally improves with more data (sensitive to "sample complexity"), the P-encoder is more sensitive to "corpus coverage" and diversity.
This led to the discovery of the SQuAD Paradox: while most datasets benefit from more training samples, the SQuAD passage encoder's performance actually peaked at just 40% of the training data and declined thereafter. The authors identified that SQuAD has high "positive passage overlap," where many different questions point to the same few passages. Over-training on these samples causes the P-encoder to become biased toward a small subset of the 21-million-passage Wikipedia corpus, hurting its generalizability.
By leveraging this insight, the authors successfully trained a P-encoder using 60% less data (only 40% of the original SQuAD set). This lean encoder achieved a top-20 retrieval accuracy of 62.3%, actually outperforming the version trained on 100% of the data (61.1%).
Conclusion
Encoder marginalization provides a powerful diagnostic tool for bi-encoder retrieval systems. By disentangling the contributions of the question and passage encoders, researchers can identify specific bottlenecks—such as passage overlap bias—and develop more data-efficient training regimes. Future work may focus on developing question-agnostic passage encoders that can generalize better to zero-shot scenarios where no specific training data is available.
Summary updated