Improving Document Representations by Generating Pseudo Query Embeddings for Dense Retrieval

Hongyin Tang, Xingwu Sun, Beihong Jin, Jingang Wang, Fuzheng Zhang, Wei Wu

TL;DR

This paper tackles the information loss problem in Bi-encoder models for dense retrieval by generating "pseudo query embeddings" for each document. These embeddings, derived from a K-means clustering of document token representations, capture diverse semantic aspects of the document, leading to improved retrieval accuracy while maintaining efficiency.

What Problem Does This Paper Address?

The first stage of document retrieval is crucial for efficiently narrowing down a large corpus to a relevant subset. While traditional sparse methods like BM25 have limitations, dense retrieval models, particularly Bi-encoders, offer better semantic understanding. However, standard Bi-encoders encode queries and documents independently. This "agnostic" encoding of documents can lead to significant information loss, especially for long documents with multiple topics, as the encoder doesn't know which specific aspects of the document a query might be interested in. This paper aims to mitigate this information loss by enriching document representations without sacrificing the efficiency of Bi-encoders.

Methods

The core innovation is the generation of pseudo query embeddings for each document. This process involves:

  1. Document Encoding: Each document is first encoded into token embeddings using a BERT-based Bi-encoder.
  2. Clustering: A K-means clustering algorithm is applied to these token embeddings. The centroids of these clusters are then treated as the "pseudo query embeddings." The intuition is that each centroid represents a distinct semantic fragment or "topic" within the document, mimicking potential queries.
  3. Inference-time Aggregation: During retrieval, the query embedding is compared against these pseudo query embeddings. To efficiently integrate with Approximate Nearest Neighbor (ANN) search libraries, a modified aggregation strategy is employed. Instead of a direct softmax over all pseudo query embeddings, the paper proposes a two-step process:
  4. An initial filtering step using an argmax operation to quickly identify promising candidate documents.
  5. A more accurate score calculation using a weighted sum of pseudo query embeddings, where weights are determined by a softmax normalization of the query's similarity to each pseudo query embedding.

This approach allows documents to be represented by multiple semantic facets, improving their ability to match diverse queries.

Datasets

The authors evaluated their method on two benchmark datasets:

  • MS MARCO: A large-scale dataset for ad-hoc text retrieval, used for both document ranking and passage ranking tasks.
  • OpenQA: A dataset designed for open-domain question answering, comprising documents from Wikipedia and question-answer pairs from various QA datasets (Natural Questions, Trivia QA, SQUAD).

Key Insights and Ideas

  • Mimicking Queries: The idea of generating "pseudo queries" from the document itself is a clever way to address the query-agnostic nature of Bi-encoders. It allows the document representation to be more context-aware without needing the actual query during the initial encoding phase.
  • Clustering for Semantic Diversity: K-means clustering effectively breaks down a long document into distinct semantic clusters, ensuring that multiple facets of the document are captured. This is crucial because a single query might only be relevant to a specific part of a long document.
  • Efficiency-Accuracy Trade-off: The proposed method achieves a good balance. By pre-computing pseudo query embeddings offline, the online retrieval process remains efficient, comparable to simpler Bi-encoder models. The modified aggregation strategy further optimizes this for ANN search.
  • Gradient Analysis: The paper provides an in-depth analysis of the loss function's gradient, showing how the pseudo query embeddings lead to more targeted updates during training, effectively pushing document embeddings closer to relevant query embeddings.

Key Results

  • MS MARCO: The proposed model achieved state-of-the-art results on the MS MARCO passage ranking task, outperforming models like DocT5Query, RepBERT, and ANCE. It also showed strong performance on the document ranking task, outperforming other models by a significant margin.
  • OpenQA: On the OpenQA dataset, the model demonstrated superior performance compared to the standard DPR model, particularly on the SQUAD dataset, suggesting its effectiveness in scenarios where documents might correspond to a higher number of distinct questions.
  • Efficiency: The model's online retrieval time is comparable to independent aggregators and significantly faster than late-interaction aggregators, confirming its efficiency.

Limitations and Future Work

  • K-means Optimization: The paper notes that the K-means implementation, while effective, is not highly parallel-friendly and could be optimized for faster computation.
  • Cluster Number: While the study explored different numbers of clusters (k), the optimal value might vary across datasets. The paper suggests that documents might not always have a large number of distinct clusters, leading to performance drops with very high k.
  • Future Work: The authors suggest exploring more sophisticated clustering algorithms and investigating how to further optimize the aggregation and scoring mechanisms for ANN search. They also hint at the potential for this approach to be applied to other dense retrieval architectures.

Summary updated

BibTeX
@inproceedings{tang2021improving,
  author = {Hongyin Tang and Xingwu Sun and Beihong Jin and Jingang Wang and Fuzheng Zhang and Wei Wu},
  title = {{Improving Document Representations by Generating Pseudo Query Embeddings for Dense Retrieval}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
  url = {https://openreview.net/forum?id=aJaNk_aGtg},
}
Figure