Back to Basics: A Simple Recipe for Improving Out-of-Domain Retrieval in Dense Encoders

Hyunji Lee, Luca Soldaini, Arman Cohan, Minjoon Seo, Kyle LoarXiv 2023 · DOI · arXiv

Introduction

Dense retrieval models, which utilize neural encoders like BERT to map text into vector embeddings, have significantly advanced search technology. However, these models often suffer from a "generalization gap": while they excel on training datasets like MSMARCO, their performance frequently collapses when applied to "out-of-domain" (OOD) topics such as medical records or technical documentation.

While the common response to this issue is to increase data and model size, this research investigates whether standard training recipes are fundamentally flawed. The authors suggest that practices designed to maximize in-domain performance often lead to overfitting, making models "brittle" when encountering new data distributions.

Methodology

The researchers systematically deconstructed the training process to identify a simpler, more robust recipe for generalization. They focused on three primary levers:

  1. Parameter-Efficient Fine-Tuning (PEFT): Specifically LoRA (Low-Rank Adaptation). Instead of updating all parameters (Full Fine-Tuning), LoRA trains only approximately 0.25% of the weights.
  2. Batch Design: Comparing in-batch negatives (random samples within a batch) against mined hard negatives (documents that are algorithmically chosen for their similarity to the query but are factually incorrect).
  3. Architectural Validation: Testing these strategies across Asymmetric dual encoders, Symmetric dual encoders, and Late Interaction models (e.g., ColBERT).

Models were trained on standard supervised datasets (MSMARCO and Google Natural Questions) and evaluated using the BEIR benchmark, which comprises 14 diverse OOD datasets.

Key Results and Insights

The study reveals that "less is more" when it comes to building robust retrievers. LoRA consistently outperformed full fine-tuning (FT) on OOD tasks, providing absolute improvements of 1.6 to 4.6 points.

Detailed performance comparison across 14 OOD datasets for Asymmetric and Symmetric Dual Encoders.

As shown in the detailed results above, LoRA's restricted parameter space acts as a natural regularizer. It prevents the model from "memorizing" the specific linguistic patterns of the training set, forcing it to rely on more general features learned during pre-training.

Conversely, the use of mined hard negatives—a standard industry practice—was found to be a double-edged sword. While beneficial for in-domain performance, it often penalized OOD generalization by encouraging the model to over-specialize on training-specific "tricks."

Summary of average OOD performance and relative differences between Full Fine-Tuning (FT) and LoRA.

The summary table highlights the significant relative gains achieved by LoRA, particularly when combined with simpler negative sampling. For instance, in asymmetric encoders, incorporating mined hard negatives during FT training dropped OOD performance by as much as 10.7%.

Efficiency and Scalability

Beyond performance gains, LoRA offers significant computational advantages. Because it requires substantially less GPU memory than full fine-tuning, researchers can utilize much larger batch sizes on the same hardware. The authors found that larger batches—providing more random in-batch negatives—consistently benefited OOD generalization, with a 4x increase in batch size improving performance from 39.0 to 40.2 nDCG@10.

Conclusion

This research provides a clear, simple recipe for improving dense retrieval: prioritize parameter-efficient tuning like LoRA and favor large batches of random negatives over complex hard-negative mining. These findings suggest that robustness in the real world comes not from squeezing every drop of performance out of a specific training set, but from preserving the general knowledge inherent in pre-trained models.

Summary updated

Abstract

Prevailing research practice today often relies on training dense retrievers on existing large datasets such as MSMARCO and then experimenting with ways to improve zero-shot generalization capabilities to unseen domains. While prior work has tackled this challenge through resource-intensive steps such as data augmentation, architectural modifications, increasing model size, or even further base model pretraining, comparatively little investigation has examined whether the training procedures themselves can be improved to yield better generalization capabilities in the resulting models. In this work, we recommend a simple recipe for training dense encoders: Train on MSMARCO with parameter-efficient methods, such as LoRA, and opt for using in-batch negatives unless given well-constructed hard negatives. We validate these recommendations using the BEIR benchmark and find results are persistent across choice of dense encoder and base model size and are complementary to other resource-intensive strategies for out-of-domain generalization such as architectural modifications or additional pretraining. We hope that this thorough and impartial study around various training techniques, which augments other resource-intensive methods, offers practical insights for developing a dense retrieval model that effectively generalizes, even when trained on a single dataset.
BibTeX
@misc{lee2023back,
  author = {Hyunji Lee and Luca Soldaini and Arman Cohan and Minjoon Seo and Kyle Lo},
  title = {{Back to Basics: A Simple Recipe for Improving Out-of-Domain Retrieval in Dense Encoders}},
  year = {2023},
  eprint = {2311.09765},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2311.09765},
}
Figure