PAIR: Leveraging Passage-Centric Similarity Relation for Improving Dense Passage Retrieval

Ruiyang Ren, Shangwen Lv, Yingqi Qu, Jing Liu, Wayne Xin Zhao, Qiaoqiao She, Hua Wu, Haifeng Wang, Ji-Rong Wen

TL;DR

Standard dense passage retrieval models only enforce that a query is closer to a positive passage than to a negative passage, which often leaves positive and negative passages clustered dangerously close together. PAIR introduces passage-centric similarity relations—requiring a positive passage to be closer to its query than to irrelevant passages—implemented via shared dual-encoder parameters, cross-encoder distillation, and a two-stage training curriculum. This structural regularizer drives hard negatives away and achieves state-of-the-art retrieval accuracy on MSMARCO and Natural Questions.

What Problem Does This Paper Address?

Dense passage retrieval relies on dual-encoders that project queries and text passages into a continuous embedding space, using dot-product similarity to rank relevance. Traditionally, these models optimize a query-centric objective: given a query $q$, maximize similarity $s(q, p^+)$ to relevant passages while minimizing $s(q, p^-)$ to irrelevant ones ($s(q, p^+) > s(q, p^-)$).

The authors point out a subtle flaw in this formulation: it completely ignores passage-to-passage relationships. Because the loss only constrains distances relative to $q$, a true positive passage $p^+$ and a topical but incorrect distractor $p^-$ can end up virtually adjacent in vector space. As a result, when a slightly different query is issued, the retriever easily confuses the two.

Methods

PAIR tackles this geometric problem with three key techniques:

               [Query-Centric]
               s(q, p+) > s(q, p-)
                   q
                 /   \
               p+ --- p-
               [Passage-Centric]
               s(p+, q) > s(p+, p-)
  1. Passage-Centric Loss & Shared Encoders:
  2. In addition to the query-centric loss ($L_Q$), PAIR introduces a passage-centric relation constraint: $s(p^+, q) > s(p^+, p^-)$. The passage-centric loss ($L_P$) normalizes scores across the positive query and negative passages relative to $p^+$.
  3. To make query–passage similarity directly comparable to passage–passage similarity, PAIR uses a single shared encoder for both queries and passages ($E_Q(\cdot) = E_P(\cdot)$) rather than separate encoders.
  4. Pseudo-Labeling via Knowledge Distillation:
  5. Hard negatives often contain unlabeled positive passages, and passage-centric constraints demand abundant, clean supervision. The authors train an expressive cross-encoder "teacher" model.
  6. For an unlabeled query pool, candidates retrieved via DPR are scored by the teacher. Confident pairs are selected using strict score thresholds ($s_{\text{pos}} = 0.9$ and $s_{\text{neg}} = 0.1$) to curate high-precision pseudo-labels.
  7. Two-Stage Training Curriculum:
  8. Because passage-centric similarity is an auxiliary constraint rather than the ultimate inference objective, multi-task interference can occur.
  9. Stage 1 (Pre-training): The shared dual-encoder optimizes the joint loss $L = (1-\alpha)L_Q + \alpha L_P$ (with $\alpha = 0.1$) using large-scale pseudo-labeled data.
  10. Stage 2 (Fine-tuning): The model is fine-tuned strictly on the target task loss ($L_Q$) using ground-truth and pseudo-labeled pairs from the target dataset.

Datasets

The authors evaluated PAIR on two standard retrieval benchmarks:

  • MSMARCO Passage Ranking: ~503k training queries, 6,980 dev queries, and 8.8M passages sampled from Bing search logs.
  • Natural Questions (NQ): ~59k training queries, 3,610 test queries, and 21.0M Wikipedia passages based on real Google search queries.

For auxiliary pre-training, the team aggregated roughly 1.8 million unlabeled queries from Yahoo! Answers, ORCAS, SQuAD, TriviaQA, and HotpotQA, pairing them with the passage corpora of MSMARCO and NQ.

Key Insights and Ideas

  • Passage-level geometry matters: Without passage-centric regularizers, the average test dot product $s(p^+, p^-)$ is actually higher than $s(p^+, q)$. Incorporating the passage-centric loss flips this ordering, pushing distractors away from the positive cluster.
  • Distinguishing semantically near-misses: Case studies highlight that PAIR avoids confusing distinct entities sharing heavy lexical overlap (e.g., distinguishing passages about "H1N1" from "H5N1", or "gall bladder" from "urinary bladder").
  • Stage separation is crucial: Ablation shows that retaining $L_P$ during final fine-tuning slightly reduces performance ($74.9 \to 74.6$ Recall@5). Treating passage-centric similarity as a pre-training prior prevents multi-task objective conflict while retaining structural representation benefits.

Key Results

PAIR established state-of-the-art results on both benchmark datasets:

  • MSMARCO Dev:
  • MRR@10: 37.9 (vs. RocketQA's 37.0, ColBERT's 36.0, and ANCE's 33.0).
  • Recall@50: 86.4% and Recall@1000: 98.2%.
  • Natural Questions Test:
  • Recall@5: 74.9% (vs. RocketQA's 74.0% and NPRINC's 73.3%).
  • Recall@20: 83.5% (vs. DPR's 78.4% and ANCE's 81.9%).
  • Recall@100: 89.1% (vs. DPR's 85.4%).

Ablations confirmed that removing passage-centric relations (w/o PSR) dropped Recall@5 from 74.9% to 73.6%, while omitting knowledge distillation (w/o KD) caused an even steeper drop to 70.9%.

Limitations and Future Work

  • Training Complexity: The pipeline involves training an expensive cross-encoder teacher, running large-scale multi-threshold distillation, and executing a two-stage training scheme.
  • Corpus Coverage Limits: As acknowledged in the ethical impact statement, static corpora like Wikipedia or MSMARCO inherently fail to capture the full breadth and evolution of real-world user queries.
  • Future Directions: The authors suggest extending passage-centric ranking mechanics to downstream tasks such as end-to-end question answering and re-ranking, and exploring more principled ranking loss formulations.

Summary updated

Abstract

Recently, dense passage retrieval has become a mainstream approach to finding relevant information in various natural language processing tasks. A number of studies have been devoted to improving the widely adopted dual-encoder architecture. However, most of the previous studies only consider query-centric similarity relation when learning the dual-encoder retriever. In order to capture more comprehensive similarity relations, we propose a novel approach that leverages both query-centric and PAssage-centric sImilarity Relations (called PAIR) for dense passage retrieval. To implement our approach, we make three major technical contributions by introducing formal formulations of the two kinds of similarity relations, generating high-quality pseudo labeled data via knowledge distillation, and designing an effective two-stage training procedure that incorporates passage-centric similarity relation constraint. Extensive experiments show that our approach significantly outperforms previous state-of-the-art models on both MSMARCO and Natural Questions datasets.
BibTeX
@inproceedings{ren2021pair,
  author = {Ruiyang Ren and Shangwen Lv and Yingqi Qu and Jing Liu and Wayne Xin Zhao and Qiaoqiao She and Hua Wu and Haifeng Wang and Ji-Rong Wen},
  title = {{PAIR: Leveraging Passage-Centric Similarity Relation for Improving Dense Passage Retrieval}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
  url = {https://openreview.net/forum?id=HuOric-h3i},
}

Loading related content...

Figure