GNN-encoder: Learning a Dual-encoder Architecture via Graph Neural Networks for Dense Passage Retrieval

Jiduan Liu, Jiahao Liu, Yang Yang, Jingang Wang, Wei Wu, Dongyan Zhao, Rui Yan

Introduction

Dense passage retrieval (DPR) is a cornerstone of modern search systems, typically relying on a two-stage process: recall and re-ranking. While dual-encoders are efficient for recall due to pre-computable embeddings, they lack the deep semantic interaction of cross-encoders. The GNN-encoder addresses this by using Graph Neural Networks (GNNs) to inject query-passage interaction wisdom into passage embeddings during the offline indexing phase, maintaining the online efficiency of dual-encoders while achieving state-of-the-art accuracy.

The Problem: Efficiency vs. Interaction

In the recall stage, dual-encoders represent queries and passages as independent vectors. This independence allows for fast similarity searches but prevents the model from capturing complex relationships between the query and the document. Conversely, cross-encoders process query-passage pairs simultaneously, offering high precision but prohibitive computational costs for large-scale retrieval. The GNN-encoder aims to bridge this gap by performing "interaction fusion" offline.

Proposed Method: GNN-encoder

The GNN-encoder treats the retrieval corpus as a massive graph where nodes represent queries and passages.

Figure 1: Architecture of the GNN-encoder, showing the fusion of query-passage interactions into embeddings via a GNN.

Graph Construction and Information Propagation

  1. Graph Construction: Edges are established between a query and its top-$k$ retrieved passages.
  2. Feature Initialization: Node features come from pre-trained dual-encoders, while edge features are initialized with cross-encoder scores to quantify interaction strength.
  3. Graph Attention Networks (GAT): Passage embeddings are updated by aggregating information from neighboring queries. This process effectively "fuses" query-side context into the passage representation.
  4. Filter Gate: A gating mechanism controls the integration of GNN-derived information, filtering out noise from irrelevant or "hard negative" queries.

Masked Graph Training (MGT)

To prevent the model from simply memorizing training pairs—a phenomenon known as information leakage—the authors introduced Masked Graph Training. By splitting queries into separate groups for graph construction and model training, the GNN-encoder is forced to learn generalizable representations rather than overfitting to a specific graph topology.

Experimental Results

The model was evaluated on major benchmarks including MSMARCO (Bing search logs), Natural Questions (Google search), and TriviaQA.

Figure 5: Main retrieval results on MSMARCO and Natural Questions benchmarks.

Key Findings

  • State-of-the-Art Performance: On MSMARCO, the GNN-encoder achieved an MRR@10 of 39.3, outperforming strong baselines like RocketQAv2 (38.8) and coCondenser (38.2). Similar gains were observed on Natural Questions with a Recall@5 of 76.8.
  • Production-Ready Efficiency: Crucially, the GNN processing occurs during offline indexing. The online retrieval latency remains identical to standard dual-encoders (~1.9ms), making it highly suitable for real-world applications.
  • Ablation Insights: The Filter Gate and MGT were found to be essential for performance, preventing noise accumulation and ensuring robust generalization to unseen queries.

Conclusion and Future Work

The GNN-encoder successfully demonstrates that graph-based interaction fusion can significantly enhance dense retrieval without sacrificing speed. While building and maintaining the graph requires substantial memory, the performance gains are substantial. Future research may explore dynamic graph updates and applying similar GNN-based refinement to multi-modal retrieval tasks.

Summary updated

Abstract

Recently, retrieval models based on dense representations are dominant in passage retrieval tasks, due to their outstanding ability in terms of capturing semantics of input text compared to the traditional sparse vector space models. A common practice of dense retrieval models is to exploit a dual-encoder architecture to represent a query and a passage independently. Though efficient, such a structure loses interaction between the query-passage pair, resulting in inferior accuracy. To enhance the performance of dense retrieval models without loss of efficiency, we propose a GNN-encoder model in which query (passage) information is fused into passage (query) representations via graph neural networks that are constructed by queries and their top retrieved passages. By this means, we maintain a dual-encoder structure, and retain some interaction information between query-passage pairs in their representations, which enables us to achieve both efficiency and efficacy in passage retrieval. Evaluation results indicate that our method significantly outperforms the existing models on MSMARCO, Natural Questions and TriviaQA datasets, and achieves the new state-of-the-art on these datasets.
BibTeX
@inproceedings{liu2022gnn,
  author = {Jiduan Liu and Jiahao Liu and Yang Yang and Jingang Wang and Wei Wu and Dongyan Zhao and Rui Yan},
  title = {{GNN-encoder: Learning a Dual-encoder Architecture via Graph Neural Networks for Dense Passage Retrieval}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  doi = {10.18653/v1/2022.findings-emnlp.39},
}
Figure