Abstract
Pre-trained Language Models have recently emerged in Information Retrieval as providing the backbone of a new generation of neural systems that outperform traditional methods on a variety of tasks. However, it is still unclear to what extent such approaches generalize in zero-shot conditions. The recent BEIR benchmark provides partial answers to this question by comparing models on datasets and tasks that differ from the training conditions. We aim to address the same question by comparing models under more explicit distribution shifts. To this end, we build three query-based distribution shifts within MS MARCO (query-semantic, query-intent, query-length), which are used to evaluate the three main families of neural retrievers based on BERT: sparse, dense, and late-interaction -- as well as a monoBERT re-ranker. We further analyse the performance drops between the train and test query distributions. In particular, we experiment with two generalization indicators: the first one based on train/test query vocabulary overlap, and the second based on representations of a trained bi-encoder. Intuitively, those indicators verify that the further away the test set is from the train one, the worse the drop in performance. We also show that models respond differently to the shifts -- dense approaches being the most impacted. Overall, our study demonstrates that it is possible to design more controllable distribution shifts as a tool to better understand generalization of IR models. Finally, we release the MS MARCO query subsets, which provide an additional resource to benchmark zero-shot transfer in Information Retrieval.
Introduction
In recent years, Pre-trained Language Models (PLMs) like BERT have revolutionized Information Retrieval (IR), consistently outperforming traditional methods like BM25 on major benchmarks. However, these models often suffer from a "generalization gap"—they perform brilliantly on data similar to their training set but stumble in "zero-shot" scenarios involving new domains or tasks. While benchmarks like BEIR evaluate zero-shot capabilities across different datasets, they often lack the granularity to isolate which specific factors (e.g., a change in topic vs. a change in query intent) cause a model to fail. This work introduces MS-Shift, a framework for creating controllable distribution shifts within the MS MARCO dataset to systematically evaluate the fundamental strengths and weaknesses of different neural architectures.
Methods and Datasets
The researchers partitioned the massive MS MARCO passage dataset into subsets to simulate three distinct types of distribution shifts:
- Semantic (Query Topics): Queries were clustered into five distinct topics (e.g., Medical, Biology/Physics, Names/Public Figures) using DistilBERT embeddings and k-means clustering.
- Intent (WH-Words): Queries were grouped by their starting words (e.g., "What" for definitions, "How" for instructions) to test generalization across different information needs.
- Syntactic (Query Length): Queries were split into "short" and "long" based on the median word count.
The evaluation employs a leave-one-out procedure: models are trained on all clusters except one and then tested on the "unseen" cluster to measure the performance drop, termed Relative Loss. To predict these drops without needing labeled ground truth, the authors introduced two "generalization indicators": Weighted Jaccard Similarity (measuring vocabulary overlap) and a Model-based Similarity (using bi-encoder representations).
Key Insights and Results
The study evaluated three families of BERT-based neural retrievers—sparse (SPLADE), dense (bi-encoders), and late-interaction (ColBERT)—alongside a monoBERT re-ranker.
Fragility vs. Robustness
- Dense Models are Fragile: Dense bi-encoders, which represent queries and documents as single vectors, were the most sensitive to shifts. In semantic experiments, they saw a relative loss of up to 18%.
- Interaction is Key: Models that allow for "late interaction" between query and document terms (ColBERT) or cross-encoders (monoBERT) were remarkably stable. ColBERT’s loss was often below 4%, maintaining high performance even when test queries differed significantly from training data.
- SPLADE’s Resilience: SPLADE, a sparse bi-encoder, proved more robust than dense models. Because it relies more on lexical components, it transfers better when vocabulary distributions are closer.
Predicting Failure
The authors demonstrated that it is possible to estimate how much a model will struggle on a new distribution by measuring the "distance" between training and test queries. There is a direct correlation between vocabulary overlap (Jaccard similarity) and model performance: as similarity increases, the relative loss in retrieval effectiveness (MRR@10) decreases.
Conclusion
MS-Shift demonstrates that controllable distribution shifts are a powerful tool for understanding the generalization limits of IR models. While interaction-based models show remarkable robustness, dense models require further investigation into training techniques like knowledge distillation or hard negative mining to close the generalization gap. The released MS MARCO query subsets provide a new resource for the community to benchmark zero-shot transfer in a controlled environment.
Summary updated