Improving Document Representations by Generating Pseudo Query Embeddings for Dense Retrieval

Hongyin Tang, Xingwu Sun, Beihong Jin, Jingang Wang, Fuzheng Zhang, Wei Wu

Abstract

Recently, the retrieval models based on dense representations have been gradually applied in the first stage of the document retrieval tasks, showing better performance than traditional sparse vector space models. To obtain high efficiency, the basic structure of these models is Bi-encoder in most cases. However, this simple structure may cause serious information loss during the encoding of documents since the queries are agnostic. To address this problem, we design a method to mimic the queries on each of the documents by an iterative clustering process and represent the documents by multiple pseudo queries (i.e., the cluster centroids). To boost the retrieval process using approximate nearest neighbor search library, we also optimize the matching function with a two-step score calculation procedure. Experimental results on several popular ranking and QA datasets show that our model can achieve state-of-the-art results.

Introduction

Modern information retrieval systems often navigate a trade-off between accuracy and efficiency. Cross-encoders (e.g., BERT-based) achieve high precision by modeling query-document interactions directly but are computationally prohibitive for large-scale retrieval. Conversely, Bi-encoders enable efficient search via pre-computed document vectors and Approximate Nearest Neighbor (ANN) libraries but suffer from a "query-agnostic" limitation. Because documents are encoded without knowledge of the query, a single vector often fails to capture the diverse semantic "facets" of a document, leading to significant information loss.

Methodology: Semi-Interactive Aggregation

The proposed approach introduces a "semi-interactive" architecture that bridges the gap between Bi-encoders and late-interaction models like ColBERT. Instead of a single vector, each document is represented by a fixed number of pseudo query embeddings generated via K-means clustering of token-level BERT embeddings.

Comparison of different aggregator architectures, highlighting the proposed semi-interactive approach (d).

The process involves:

  1. Encoding: Generating token-level embeddings using a BERT backbone.
  2. Clustering: Applying K-means to these embeddings to produce $k$ cluster centroids, which act as "pseudo queries" representing different semantic fragments of the document.
  3. Query-Aware Aggregation: At inference, the model computes similarity between the actual query and these centroids, using a softmax-weighted sum to create a dynamic, query-aware document representation.
  4. ANN Optimization: To maintain efficiency, a two-step scoring procedure is used: an initial argmax (top-1 centroid) search for candidate retrieval via Faiss, followed by full softmax aggregation on the top-R candidates.

Experimental Results

The model was evaluated on MS MARCO (Passage and Document Ranking) and OpenQA benchmarks (Natural Questions, Trivia QA, SQuAD).

Performance comparison on MS MARCO Document Ranking showing the model achieving state-of-the-art MRR.

As shown in the results for MS MARCO, the proposed model achieved a state-of-the-art 39.2 MRR@100, outperforming established baselines like ANCE. In OpenQA tasks, it demonstrated significant gains, such as an 88.6 Top-100 accuracy on SQuAD, nearly 4 points higher than the standard Dense Passage Retriever (DPR). Furthermore, the method proved to be approximately 5x faster than late-interaction models during retrieval (180ms vs 940ms).

Discussion and Insights

The core innovation lies in mimicking query-document interaction by pre-calculating representative document fragments. This "semi-interactive" design offers several advantages:

  • Efficiency: By storing only $k$ centroids (e.g., 4 or 8) rather than every token embedding, the model significantly reduces storage and computational complexity compared to models like ColBERT.
  • Gradient Dynamics: Analysis suggests that cluster centroids help convergence by amplifying query-relevant gradients during training, preventing semantic dilution.

Limitations

Despite its performance, the model currently uses a fixed number of clusters ($k$) for all documents, which may not account for varying document complexity. Additionally, while more efficient than late-interaction models, it still incurs higher storage overhead than standard single-vector Bi-encoders. Future work may explore adaptive clustering and methods to handle documents exceeding the 512-token BERT limit.

Summary updated

Abstract

Recently, the retrieval models based on dense representations have been gradually applied in the first stage of the document retrieval tasks, showing better performance than traditional sparse vector space models. To obtain high efficiency, the basic structure of these models is Bi-encoder in most cases. However, this simple structure may cause serious information loss during the encoding of documents since the queries are agnostic. To address this problem, we design a method to mimic the queries on each of the documents by an iterative clustering process and represent the documents by multiple pseudo queries (i.e., the cluster centroids). To boost the retrieval process using approximate nearest neighbor search library, we also optimize the matching function with a two-step score calculation procedure. Experimental results on several popular ranking and QA datasets show that our model can achieve state-of-the-art results.
BibTeX
@inproceedings{tang2021improving,
  author = {Hongyin Tang and Xingwu Sun and Beihong Jin and Jingang Wang and Fuzheng Zhang and Wei Wu},
  title = {{Improving Document Representations by Generating Pseudo Query Embeddings for Dense Retrieval}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
  url = {https://openreview.net/forum?id=aJaNk_aGtg},
}

Loading related content...

Figure