Unsupervised Dense Retrieval for Scientific Articles

Dan Li, Vikrant Yadav, Zubair Afzal, George TsatsaronisEMNLP 2022

TL;DR

This paper introduces an unsupervised dense retrieval system for scientific articles, overcoming the challenge of scarce labeled data by employing a novel generative pseudo-labeling approach. Their system significantly outperforms traditional lexical search and existing dense retrieval models, demonstrating the effectiveness of domain-specific unsupervised training.

What Problem Does This Paper Address?

Traditional search engines for scientific literature, like those powering platforms such as ScienceDirect, rely on lexical search methods (e.g., BM25). While simple to implement, these methods suffer from the "lexical gap" problem: they struggle with synonyms, misspellings, ambiguous terms, and word order, leading to suboptimal search results. Dense retrieval (DR) models offer a promising solution by mapping queries and documents into a shared vector space, enabling semantic understanding. However, applying state-of-the-art DR models in industrial settings is hindered by the lack of labeled data for training and evaluation, especially for specialized domains like scientific articles. Furthermore, existing DR models trained on general datasets (like MS-MARCO) often fail to generalize well to new domains due to differences in data distribution and domain-specific language.

Methods

The core of this work is the application and adaptation of the Generative Pseudo Labeling (GPL) model for unsupervised dense retrieval. The GPL approach tackles the lack of labeled data through a three-step process:

  1. Query Generation: A pre-trained query generator (all-t5-base-v1) is used to create synthetic queries from existing passages. These passage-query pairs serve as pseudo positive examples.
  2. Negative Sampling: To generate pseudo negative examples, the authors use existing dense retrieval models (msmarco-distilbert-base-v3 and msmarco-MiniLM-L-6-v3) to retrieve passages that are similar to the generated queries. The top 50 retrieved passages from each model are considered negatives.
  3. Knowledge Distillation: A student-teacher training paradigm is employed. The "teacher" is a powerful cross-encoder model (ms-marco-MiniLM-L-6-v2) trained on MS-MARCO, known for its strong generalization. The "student" is the bi-encoder DR model being trained. A MarginMSE loss is used to distill knowledge from the teacher to the student, which is more robust to noisy pseudo labels than traditional ranking losses. This loss encourages the bi-encoder to produce similar embeddings for positive pairs and dissimilar ones for negative pairs, but with a margin that accounts for label noise.

To address domain imbalance, the authors specifically construct a balanced training dataset by sampling passages from multiple domains to ensure better generalization.

Datasets

  • Training Data: The authors utilize a large corpus of scientific articles from Elsevier. They construct a training set of approximately 4 million examples, derived from 83,000 passages. They also experiment with a balanced subset of these passages, sampled evenly across 20 scientific domains.
  • Evaluation Datasets: Two test sets were created:
  • Manual Test Set: A smaller, high-quality set of 20 queries and 539 query-passage pairs, manually annotated by three workers.
  • Auto Test Set: A larger set of 200 queries and 2614 relevant passages, automatically constructed using meta-information from the corpus. This set is noted to have a higher proportion of lexically relevant passages.

Key Insights and Ideas

  • Unsupervised DR is Viable for Specialized Domains: The paper demonstrates that even without explicit human-labeled data, dense retrieval can be effectively trained for specialized domains like scientific literature.
  • Pseudo-Labeling is Crucial: The success hinges on the generative pseudo-labeling approach, which creates a large volume of training signals from unlabeled data.
  • Domain Balancing Matters: Training on a balanced distribution of passages across different scientific domains significantly improves retrieval performance compared to using an unbalanced, naturally occurring distribution. This highlights the impact of domain shift on DR models.
  • Knowledge Distillation from Cross-Encoders is Effective: Using a strong cross-encoder as a teacher model for knowledge distillation is a robust strategy for training a more efficient bi-encoder, especially when dealing with noisy pseudo labels.
  • Lexical Search Still Strong, but DR Can Surpass: While BM25 (lexical search) performs robustly, the proposed GPL model, especially when trained on balanced data, shows significant improvements, particularly in metrics like MRR and R@100.

Key Results

  • The proposed unsupervised dense retrieval model (GPL) significantly improves retrieval performance compared to the current production lexical search (BM25) and a zero-shot dense retrieval baseline (TAS).
  • On the Manual test set, the best performing model, GPL_BLC (GPL trained on balanced data), achieved an MRR@10 of 87.62 and an R@100 of 91.77, outperforming BM25 and TAS.
  • The BM25+CE (two-stage BM25 with cross-encoder re-ranking) served as an upper bound, achieving the highest scores (e.g., NDCG@10 of 84.90 on the Manual test set), indicating the potential of dense retrieval components.
  • Training with approximately 1 million examples was found to be sufficient for good performance, suggesting that using the entire corpus for training might not be strictly necessary.
  • A balanced training corpus led to a noticeable improvement in NDCG@10, especially for queries that initially had low scores.

Limitations and Future Work

  • Query Generator Domain Mismatch: The query generator was trained on a different domain, potentially leading to missed keywords or phrases relevant to the scientific article domain.
  • Negative Retriever Adaptation: The negative retrievers used were not specifically adapted to the scientific article domain, meaning they might not generate "hard negatives" (i.e., passages that are semantically close but incorrect).
  • Production Evaluation: The semantic search engine has not yet been evaluated on a production population.

Future work includes:

  • Training the query generator and negative retrievers on the target domain data to improve pseudo-label quality.
  • Conducting online evaluations with actual users to assess real-world performance.
  • Investigating the impact of splitting longer passages into shorter segments, as current passages are truncated.

Summary updated

BibTeX
@inproceedings{li2022unsupervised,
  author = {Dan Li and Vikrant Yadav and Zubair Afzal and George Tsatsaronis},
  title = {{Unsupervised Dense Retrieval for Scientific Articles}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
}
Figure