Noisy Self-Training with Synthetic Queries for Dense Retrieval

Fan Jiang, Tom Drummond, Trevor Cohn

Abstract

Information Retrieval (IR) has undergone a paradigm shift from traditional keyword matching to dense retrieval using neural dual-encoders. However, these models require vast amounts of labeled data to generalize effectively, particularly in specialized domains. This paper introduces a "noisy self-training" framework that leverages synthetic query generation and teacher-student distillation. By utilizing soft labels and injecting input noise during training, the proposed method achieves state-of-the-art results on both general and out-of-domain benchmarks without necessitating additional human-annotated data.

1. Introduction

Dense retrieval models map queries and documents into a shared vector space, allowing for semantic matching that surpasses the limitations of lexical systems like BM25. Despite their potential, these models are notoriously data-hungry. Collecting high-quality, human-labeled query-passage pairs is prohibitively expensive, creating a bottleneck for "out-of-domain" (OOD) applications such as medical or legal search.

Existing approaches attempt to mitigate this by generating synthetic queries for unlabeled text. However, synthetic queries are often ambiguous or only tangentially related to their source passages. Treating these as "hard labels"—where a query is strictly mapped to one passage—can introduce noise and incorrect patterns into the model. This research addresses these challenges by proposing a robust self-training pipeline that effectively handles the inherent noise of synthetic data.

2. Methodology

The proposed Noisy Self-Training pipeline consists of four primary stages designed to refine the model's retrieval capabilities iteratively:

  1. Query Generation: A T5-based model is trained on a small set of "gold" (human-labeled) data. This model then generates potential questions for a large corpus of unlabeled passages.
  2. Teacher Supervision: Rather than assuming a synthetic query only matches its originating passage, a "Teacher" retriever (pre-trained on gold data) scores the query against a pool of passages. This generates soft labels (logits), providing a probability distribution that reflects the Teacher's confidence in various passage matches.
  3. Noisy Student Training: A "Student" model is trained to mimic the Teacher’s scoring distribution. To ensure the Student learns robust features rather than simply memorizing the Teacher's outputs, the researchers inject input noise. This includes randomly shuffling, deleting, or masking words in both queries and passages, forcing the model to learn generalized semantic representations.
  4. Finetuning and Iteration: The Student is finally finetuned on the original gold data. This process can be iterated, where the improved Student becomes the new Teacher for the next cycle.

3. Experimental Evaluation

The framework was evaluated across several rigorous benchmarks to test both general and specialized performance:

  • General Domain: MS-MARCO (passage ranking) and Natural Questions (NQ).
  • Out-of-Domain (OOD): The BEIR benchmark, encompassing 18 diverse datasets including TREC-COVID, BioASQ, and FEVER.
  • Low-Resource Scenarios: Testing on MS-MARCO subsets ranging from 1% to 70% of the total available data.

4. Key Insights

The "Weak Relation" Problem

Synthetic queries often suffer from being "spuriously related" to their source. For instance, a query about "Manhattan Project achievements" might be generated from a passage that only mentions the word "achievement" incidentally. By using soft labels, the Teacher model can redirect the Student toward more relevant passages, preventing the reinforcement of weak or incorrect associations.

Robustness through Noise

Injecting noise during training—specifically token shuffling—effectively teaches the dense retriever to handle disorganized or keyword-heavy queries. This allows the model to capture "lexical matching" properties (identifying salient phrases) typically associated with sparse retrievers, while maintaining the semantic depth of a neural model.

Data and Computational Efficiency

The framework demonstrates remarkable data efficiency. Using only 30% of the labeled MS-MARCO training data, the noisy self-training approach matched the performance of a standard model trained on 100% of the data. Furthermore, compared to existing methods like GPL, this approach reduced training time by 3x and achieved a 25x speed-up in the labeling process.

5. Results

The "Student coCondenser" achieved a new state-of-the-art on MS-MARCO with a 39.4 MRR@10. In OOD settings, the model improved the average nDCG@10 on the BEIR benchmark by 3.0% over the teacher model. Notably, it outperformed task-specialized retrievers on 6 out of 18 BEIR tasks while remaining a single, universal model. The framework also proved beneficial for cross-encoder rerankers, yielding an additional 1.1 average point boost in OOD performance.

6. Conclusion and Future Work

Noisy self-training provides a scalable, efficient path for improving dense retrieval without the need for massive human annotation. While the pipeline requires significant compute (approximately 3 days on 4x A100 GPUs), its efficiency gains over competing methods are substantial. Future research will focus on reducing the dependency on the initial labeled data for the Teacher model and investigating ways to mitigate the propagation of social biases inherent in the language models used for query generation.

Summary updated

BibTeX
@inproceedings{jiang2023noisy,
  author = {Fan Jiang and Tom Drummond and Trevor Cohn},
  title = {{Noisy Self-Training with Synthetic Queries for Dense Retrieval}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  doi = {10.18653/v1/2023.findings-emnlp.803},
}
Figure