Benchmarking Middle-Trained Language Models for Neural Search

Hervé Déjean, Stéphane Clinchant, Carlos Lassance, Simon Lupart, Thibault Formal

Abstract

In Neural Information Retrieval (IR), "middle-trained" language models (MTLMs) such as RetroMAE and CoCondenser are designed to bridge the gap between general-purpose pre-training and specific retrieval tasks. This paper presents a rigorous, standardized comparison of these models against standard BERT baselines. The study reveals that while middle training offers clear advantages in low-resource settings, the performance gap narrows significantly when high-quality finetuning recipes—incorporating hard negatives and distillation—are employed.

Introduction

The standard pipeline for Neural IR involves finetuning a Pre-trained Language Model (PLM) like BERT for search tasks. However, BERT's original Masked Language Modeling (MLM) objective is not perfectly aligned with the requirements of query-document matching. To address this, researchers introduced Middle Training (MT): an unsupervised intermediate step that "preconditions" the network for retrieval.

While models like CoCondenser, RetroMAE, and LexMAE have claimed state-of-the-art results, they often utilize disparate finetuning "recipes" (e.g., varying numbers of negatives or distillation techniques). This inconsistency makes it difficult to determine whether performance gains stem from the model architecture or the training setup. This paper isolates the impact of middle training through a standardized experimental framework.

Experimental Methodology

The authors compared several model architectures and middle-training objectives under identical finetuning conditions:

  • Retrieval Frameworks: Both Dense (CLS token embeddings) and Sparse (SPLADE framework) retrievers.
  • Models: CoCondenser, RetroMAE (trained on Wikipedia or MS MARCO), and LexMAE.
  • Baselines: Standard BERT and "BERT-MS" (standard MLM on the MS MARCO collection).
  • Finetuning Protocols: A "low-level" setting (1 negative, no distillation) and a "high-level" setting (32 hard negatives with cross-encoder distillation).

Key Insights: The Finetuning Leveler

A primary finding is that the specific middle-training method becomes less critical as the finetuning procedure improves. For sparse models (SPLADE), advanced finetuning significantly levels the playing field, allowing even basic BERT models to remain competitive.

SPLADE performance across different finetuning settings.

Furthermore, middle training appears to function largely as a form of domain adaptation. A simple BERT model middle-trained with standard MLM on the target corpus (BERT-MS) performs remarkably well, suggesting that adapting to the specific vocabulary and statistics of the retrieval collection is a major component of MT's success. Interestingly, MT tasks designed for dense retrieval also benefit sparse models, indicating that retrieval preconditioning captures general features useful across architectures.

Performance Results

In the most optimized setting (distillation + 32 negatives), specialized models still hold an edge, particularly in dense retrieval.

Performance of various models in the most optimized setting (32 negatives + distillation).
  • Dense Retrieval: RetroMAE MS emerged as the strongest performer, achieving an MRR@10 of 39.2, compared to 37.4 for standard BERT.
  • Sparse Retrieval: The gap was tighter; LexMAE reached 39.8 MRR@10, with standard BERT following closely at 39.1.
  • Zero-Shot Evaluation: On the BEIR benchmark, middle training on MS MARCO did not consistently improve generalization. Standard BERT often matched or exceeded specialized MTLMs in zero-shot scenarios.
  • Domain Shift: Testing on the medical dataset TripClick showed that models middle-trained on MS MARCO underperformed compared to those trained on domain-specific data, reinforcing the importance of corpus-specific adaptation.

Conclusion

The study concludes that while middle training is a valuable tool for neural search, its benefits are most pronounced when finetuning resources are limited. As finetuning recipes become more sophisticated, the relative advantage of complex MT objectives diminishes. The authors advocate for more transparent benchmarking in the IR community, where finetuning recipes are shared alongside models to ensure fair and reproducible comparisons.

Summary updated

Abstract

Middle training methods aim to bridge the gap between the Masked Language Model (MLM) pre-training and the final finetuning for retrieval. Recent models such as CoCondenser, RetroMAE, and LexMAE argue that the MLM task is not sufficient enough to pre-train a transformer network for retrieval and hence propose various tasks to do so. Intrigued by those novel methods, we noticed that all these models used different finetuning protocols, making it hard to assess the benefits of middle training. We propose in this paper a benchmark of CoCondenser, RetroMAE, and LexMAE, under the same finetuning conditions. We compare both dense and sparse approaches under various finetuning protocols and middle training on different collections (MS MARCO, Wikipedia or Tripclick). We use additional middle training baselines, such as a standard MLM finetuning on the retrieval collection, optionally augmented by a CLS predicting the passage term frequency. For the sparse approach, our study reveals that there is almost no statistical difference between those methods: the more effective the finetuning procedure is, the less difference there is between those models. For the dense approach, RetroMAE using MS MARCO as middle-training collection shows excellent results in almost all the settings. Finally, we show that middle training on the retrieval collection, thus adapting the language model to it, is a critical factor. Overall, a better experimental setup should be adopted to evaluate middle training methods. Code available at https://github.com/naver/splade/tree/benchmarch-SIGIR23
BibTeX
@inproceedings{dejean2023benchmarking,
  author = {Hervé Déjean and Stéphane Clinchant and Carlos Lassance and Simon Lupart and Thibault Formal},
  title = {{Benchmarking Middle-Trained Language Models for Neural Search}},
  booktitle = {International ACM SIGIR Conference on Research and Development in Information Retrieval},
  year = {2023},
  doi = {10.1145/3539618.3591956},
}
Figure