Abstract
In Neural Information Retrieval (IR), "middle-trained" language models (MTLMs) such as RetroMAE and CoCondenser are designed to bridge the gap between general-purpose pre-training and specific retrieval tasks. This paper presents a rigorous, standardized comparison of these models against standard BERT baselines. The study reveals that while middle training offers clear advantages in low-resource settings, the performance gap narrows significantly when high-quality finetuning recipes—incorporating hard negatives and distillation—are employed.
Introduction
The standard pipeline for Neural IR involves finetuning a Pre-trained Language Model (PLM) like BERT for search tasks. However, BERT's original Masked Language Modeling (MLM) objective is not perfectly aligned with the requirements of query-document matching. To address this, researchers introduced Middle Training (MT): an unsupervised intermediate step that "preconditions" the network for retrieval.
While models like CoCondenser, RetroMAE, and LexMAE have claimed state-of-the-art results, they often utilize disparate finetuning "recipes" (e.g., varying numbers of negatives or distillation techniques). This inconsistency makes it difficult to determine whether performance gains stem from the model architecture or the training setup. This paper isolates the impact of middle training through a standardized experimental framework.
Experimental Methodology
The authors compared several model architectures and middle-training objectives under identical finetuning conditions:
- Retrieval Frameworks: Both Dense (CLS token embeddings) and Sparse (SPLADE framework) retrievers.
- Models: CoCondenser, RetroMAE (trained on Wikipedia or MS MARCO), and LexMAE.
- Baselines: Standard BERT and "BERT-MS" (standard MLM on the MS MARCO collection).
- Finetuning Protocols: A "low-level" setting (1 negative, no distillation) and a "high-level" setting (32 hard negatives with cross-encoder distillation).
Key Insights: The Finetuning Leveler
A primary finding is that the specific middle-training method becomes less critical as the finetuning procedure improves. For sparse models (SPLADE), advanced finetuning significantly levels the playing field, allowing even basic BERT models to remain competitive.
Furthermore, middle training appears to function largely as a form of domain adaptation. A simple BERT model middle-trained with standard MLM on the target corpus (BERT-MS) performs remarkably well, suggesting that adapting to the specific vocabulary and statistics of the retrieval collection is a major component of MT's success. Interestingly, MT tasks designed for dense retrieval also benefit sparse models, indicating that retrieval preconditioning captures general features useful across architectures.
Performance Results
In the most optimized setting (distillation + 32 negatives), specialized models still hold an edge, particularly in dense retrieval.
- Dense Retrieval: RetroMAE MS emerged as the strongest performer, achieving an MRR@10 of 39.2, compared to 37.4 for standard BERT.
- Sparse Retrieval: The gap was tighter; LexMAE reached 39.8 MRR@10, with standard BERT following closely at 39.1.
- Zero-Shot Evaluation: On the BEIR benchmark, middle training on MS MARCO did not consistently improve generalization. Standard BERT often matched or exceeded specialized MTLMs in zero-shot scenarios.
- Domain Shift: Testing on the medical dataset TripClick showed that models middle-trained on MS MARCO underperformed compared to those trained on domain-specific data, reinforcing the importance of corpus-specific adaptation.
Conclusion
The study concludes that while middle training is a valuable tool for neural search, its benefits are most pronounced when finetuning resources are limited. As finetuning recipes become more sophisticated, the relative advantage of complex MT objectives diminishes. The authors advocate for more transparent benchmarking in the IR community, where finetuning recipes are shared alongside models to ensure fair and reproducible comparisons.
Summary updated