Introduction
Dense passage retrieval (DPR) is a cornerstone of modern search systems, typically relying on a two-stage process: recall and re-ranking. While dual-encoders are efficient for recall due to pre-computable embeddings, they lack the deep semantic interaction of cross-encoders. The GNN-encoder addresses this by using Graph Neural Networks (GNNs) to inject query-passage interaction wisdom into passage embeddings during the offline indexing phase, maintaining the online efficiency of dual-encoders while achieving state-of-the-art accuracy.
The Problem: Efficiency vs. Interaction
In the recall stage, dual-encoders represent queries and passages as independent vectors. This independence allows for fast similarity searches but prevents the model from capturing complex relationships between the query and the document. Conversely, cross-encoders process query-passage pairs simultaneously, offering high precision but prohibitive computational costs for large-scale retrieval. The GNN-encoder aims to bridge this gap by performing "interaction fusion" offline.
Proposed Method: GNN-encoder
The GNN-encoder treats the retrieval corpus as a massive graph where nodes represent queries and passages.
Graph Construction and Information Propagation
- Graph Construction: Edges are established between a query and its top-$k$ retrieved passages.
- Feature Initialization: Node features come from pre-trained dual-encoders, while edge features are initialized with cross-encoder scores to quantify interaction strength.
- Graph Attention Networks (GAT): Passage embeddings are updated by aggregating information from neighboring queries. This process effectively "fuses" query-side context into the passage representation.
- Filter Gate: A gating mechanism controls the integration of GNN-derived information, filtering out noise from irrelevant or "hard negative" queries.
Masked Graph Training (MGT)
To prevent the model from simply memorizing training pairs—a phenomenon known as information leakage—the authors introduced Masked Graph Training. By splitting queries into separate groups for graph construction and model training, the GNN-encoder is forced to learn generalizable representations rather than overfitting to a specific graph topology.
Experimental Results
The model was evaluated on major benchmarks including MSMARCO (Bing search logs), Natural Questions (Google search), and TriviaQA.
Key Findings
- State-of-the-Art Performance: On MSMARCO, the GNN-encoder achieved an MRR@10 of 39.3, outperforming strong baselines like RocketQAv2 (38.8) and coCondenser (38.2). Similar gains were observed on Natural Questions with a Recall@5 of 76.8.
- Production-Ready Efficiency: Crucially, the GNN processing occurs during offline indexing. The online retrieval latency remains identical to standard dual-encoders (~1.9ms), making it highly suitable for real-world applications.
- Ablation Insights: The Filter Gate and MGT were found to be essential for performance, preventing noise accumulation and ensuring robust generalization to unseen queries.
Conclusion and Future Work
The GNN-encoder successfully demonstrates that graph-based interaction fusion can significantly enhance dense retrieval without sacrificing speed. While building and maintaining the graph requires substantial memory, the performance gains are substantial. Future research may explore dynamic graph updates and applying similar GNN-based refinement to multi-modal retrieval tasks.
Summary updated