Search systems usually work in two steps. A cheap first-stage retriever such as BM25 pulls a shortlist of $k$ candidate passages, and an expensive neural re-ranker reorders that shortlist. The size $k$ is called the re-ranking depth. A well-replicated and puzzling finding is that making $k$ larger often makes final rankings worse, even though a larger shortlist can only contain more relevant passages. Chinese BERT re-rankers peak at $k=70$ and decline by $k=120$ (THUIR at the NTCIR-15 WWW-3 Task). Medical question matchers decline past 95 candidates (Li et al.). Agent-driven search degrades past $k=50$ (Zero-Shot Retrieval with Search Agents and Hybrid Environments). The accepted explanation is that the re-ranker is misled by the noisy tail of the shortlist. Practitioners then choose $k$ by tuning on public benchmarks such as MS MARCO.
This note argues that the depth curve measured on public benchmarks mixes the real noise effect with two measurement artifacts that pull in opposite directions. The first is missing relevance judgments. Deep candidates were often never judged, so they are scored as irrelevant even when they are relevant, which makes deep re-ranking look worse than it is (On the Reliability of Test Collections for Evaluating Systems of Different Types). The second is benchmark contamination, meaning the re-ranker has seen the test queries or their relevant passages during training. A contaminated re-ranker can pick out memorized positives from any depth, which flattens the curve and hides noise sensitivity. We propose a controlled study that separates these three components. It plants contamination in half of the test queries of a single model, so each model serves as its own control, and it completes the judgments for deep candidates. The practical question is whether a $k$ tuned on a public benchmark is the right $k$ for fresh, unseen queries.
Core Claim
Our hypothesis is that contamination makes the depth curve flatter on leaked queries. A re-ranker that has memorized a query's relevant passage keeps it at rank one whether the shortlist holds 50 or 1,000 distractors. On queries it has never seen, the same model drops in quality as $k$ grows. If public test sets are partly leaked into the text these models were trained on, the best $k$ chosen on those sets will be too deep for fresh traffic. Missing judgments push the chosen $k$ the other way, toward being too shallow.
A simple decomposition states the claim. Let $M_{\text{obs}}(k)$ be the measured nDCG@10 at depth $k$ and $M_{\text{true}}(k)$ the value we would measure with complete judgments on uncontaminated queries. Then
$$M_{\text{obs}}(k) = M_{\text{true}}(k) - U(k) + C(k),$$
where $U(k)$ is the penalty from relevant-but-unjudged passages reaching the top ten, and $C(k)$ is the gain from memorized positives on leaked queries. Both terms plausibly grow with $k$. Which one dominates, and therefore whether public benchmarks overstate or understate the safe depth, is the empirical question. The new contribution is to estimate $U(k)$ and $C(k)$ separately as functions of depth. Prior work treats pooling bias, contamination, and depth each in isolation.
Relation to Prior Work
Four bodies of work bear on this, in different roles.
The depth papers motivate the study. Text-to-Text Multi-view Learning for Passage Re-ranking shows that adding a passage-to-query generation objective makes T5 re-rankers more robust from $k=10$ to $1000$ on MS MARCO. Its training data and its pretraining corpus (C4 crawl) may overlap with MS MARCO, and it never asks whether that matters. On the Effect of Low-Frequency Terms on Neural-IR Models attributes part of the depth cliff to rare words in pruned vocabularies. We take that as one concrete source of the real noise term $M_{\text{true}}(k)$ and keep it fixed by using subword models throughout.
Pooling-bias work constrains how we measure. On the Reliability of Test Collections for Evaluating Systems of Different Types shows that lexical-only pools on TREC 2019 Deep Learning rank neural systems almost randomly (Kendall's $\tau = 0.004$ for passage MRR). BEIR reports that re-judging TREC-COVID raised a dense retriever's score by 8 points. Neither paper relates this incompleteness to re-ranking depth. We estimate it per depth.
Contamination work supplies the method. Data Contamination: From Memorization to Exploitation injects test sets into BERT pretraining at controlled duplication counts. It finds that memorization does not always produce downstream gains, and that exploitation reaches 40% at 200 duplicates. We reuse its injection protocol, moved from single-step classification to two-stage ranking. DCR distinguishes contamination with inputs only from contamination with input-label pairs. That distinction maps cleanly onto ranking: leaking a passage's text is not the same as leaking a query paired with its relevant passage. Search-Time Data Contamination shows that leakage can also enter through what is retrieved at inference time, not only through training. It treats the retriever as a black box and does not vary depth. We keep that inference-time route as a secondary condition.
Research Program
The core is a controlled-contamination study on MS MARCO passage ranking and the TREC 2019 Deep Learning passage queries. Those 43 queries carry the deepest human judgments in the file.
We split the test queries at random into a leaked half and a clean half. Starting from BERT-base, we run continued masked-language pretraining on a fixed Wikipedia slice, injecting material from the leaked half at duplication counts of 0, 1, 10, and 100. The injected material comes in two forms. In passage-only contamination, the leaked queries' relevant passages appear alone. In paired contamination, each query appears concatenated with its relevant passage. Every pretrained variant is then fine-tuned identically on the MS MARCO training triples and used to re-rank BM25 candidates at $k \in \{10, 20, 50, 100, 200, 500, 1000\}$.
Because the leaked and clean halves are scored by the same model, the difference between their depth curves isolates $C(k)$ from model capacity and training noise. We repeat the protocol with T5-base, both single-view and with the multi-view objective of Ju et al.. This tests whether that method's depth robustness survives on clean queries.
To estimate $U(k)$, we collect human relevance judgments for every unjudged passage that any configuration places in its top ten, at any depth, on the 43 TREC DL queries. The expected volume is a few thousand passages. With these judgments, depth curves can be recomputed under complete labels.
To check $M_{\text{true}}(k)$ directly, annotators write about 50 new queries over the MS MARCO passage corpus. These queries cannot have been leaked. We judge them deeply over the union of all systems' top 20.
As a secondary condition, we place synthetic "answer-key mirror" pages in the corpus, mimicking Han et al.. Each mirror is a page that lists queries next to their relevant passage IDs and text. We then measure at what depth they enter the shortlist and whether the re-ranker promotes them.
Baselines
The naive reference point is current practice: a fixed depth of $k=100$ or $k=1000$, scored with the original TREC DL and MS MARCO dev labels, with $k$ chosen on MS MARCO dev. Every corrected estimate is compared against this. It is the number a practitioner would actually report.
The strongest existing method for depth robustness is the multi-view T5 re-ranker of Ju et al.. That paper reports MRR@10 of 0.401 on MS MARCO dev for T5-large with the generation objective, against 0.395 for the single-view model. We retrain it at base scale under our identical data, pretraining, and tuning budget, sweeping its mixing rate over the same grid the paper used. It counts as truly depth-robust only if its advantage over single-view holds on the clean half and the fresh queries. If that advantage is concentrated on leaked queries, the robustness claim is confounded.
For context on how sharp depth peaks are in practice, THUIR reports nDCG 0.4112 at $k=70$ falling to 0.3940 at $k=120$ on NTCIR-15 WWW-3. That is a different collection, so we use it only to calibrate expected effect sizes.
The key ablation is the zero-duplication model scored on both halves. It must show no gap between the halves, which validates the split. A second ablation compares passage-only and paired contamination to test whether the depth effect needs the query-passage pairing.
The proposal "beats" the naive protocol in one specific sense. The $k$ selected from corrected curves (complete judgments, clean queries only) must give higher nDCG@10 on the fresh queries than the $k$ selected on MS MARCO dev, with a paired bootstrap interval excluding zero.
Why not just…?
Why not just score with condensed lists or re-judge the unjudged passages, and forget contamination? This is a real threat to half the claim, so we run it explicitly. Re-judging is exactly how we estimate $U(k)$. If complete judgments alone make the depth curve on fresh queries match the curve on public queries, then contamination does not matter for choosing depth, and we will report that. The leaked-versus-clean split within a single model tests the remaining question directly, and pooling correction cannot answer it.
Why not just tune $k$ on a held-out split of the benchmark? A held-out split of MS MARCO comes from the same public release and is equally exposed to web crawls. Tuning on it inherits whatever $C(k)$ the test set has. Our fresh-query set exists to check this. If $k$ tuned on dev transfers to fresh queries without loss, the objection wins.
Why not just evaluate only on fresh or synthetic collections? Synthetic generators like DyVal cover formal reasoning, not open-domain relevance. Fresh collections are expensive to judge deeply, and they decay once published (Han et al.). We use a small fresh set as ground truth, but the goal is to know how much the cheap public benchmarks mislead, so that they can be corrected rather than discarded.
Why not just use a depth-robust re-ranker and stop worrying about $k$? If the multi-view model's curve were flat on clean queries, this would be the right answer. Whether it is flat on clean queries is precisely what the clean-half comparison measures.
Decisive Checks
Three results would settle the study.
First, at 100 duplications of paired contamination, the leaked half's nDCG@10 curve should decline measurably less from $k=50$ to $k=1000$ than the clean half's curve under the same model. If the curves coincide, $C(k)$ is negligible, and we would conclude, consistent with Magar and Schwartz, that memorization is not being exploited.
Second, recomputing curves with completed judgments should shift the best $k$ by a measurable amount. The direction of that shift tells us whether $U(k)$ or $C(k)$ dominates on real public benchmarks.
Third, the $k$ selected from the corrected curves must outperform the $k$ selected on dev when both are applied to the fresh queries.
Metrics throughout are nDCG@10 and MRR@10. We also report the best depth $k^*$ with bootstrap intervals, the fraction of top-ten slots occupied by unjudged passages at each $k$, and, for the mirror condition, the depth at which mirror pages first enter the top ten.
References
- THUIR at the NTCIR-15 WWW-3 Task
- Retrieving and ranking short medical questions with two stages neural matching model
- Zero-Shot Retrieval with Search Agents and Hybrid Environments
- On the Reliability of Test Collections for Evaluating Systems of Different Types
- Text-to-Text Multi-view Learning for Passage Re-ranking
- On the Effect of Low-Frequency Terms on Neural-IR Models
- BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models
- Data Contamination: From Memorization to Exploitation
- DCR: Quantifying Data Contamination in LLMs Evaluation
- Search-Time Data Contamination
- DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks