Dense retrievers turn every query and document into a single vector and rank documents by how close their vectors are to the query's. They work well on collections of tens of thousands of documents and then degrade as the collection grows, because more irrelevant documents land closer to the query than the right answer does. Meanwhile, retrieval teams increasingly check their systems with synthetic test collections: queries and relevance labels written by a large language model (LLM) instead of by people. This proposal asks whether these two facts interact badly. The concern is that LLM-written queries sit unusually close to the document they were generated from. Their correct answer would then have a wide lead over the crowd of distractors, so synthetic benchmarks would systematically underestimate how much a dense retriever loses when the index scales from thousands to millions of documents.
The proposal makes two concrete contributions. First, it measures, on nested index sizes of the same corpus, whether rankings of retrieval systems on synthetic queries track rankings on human queries as the index grows, and in particular whether the synthetic queries detect "flips," where a dense model beats BM25 on a small index and loses to it on a large one. Second, it introduces a per-query collision profile: the query's positive-document score compared against the upper tail of distractor scores. A small-index sample is enough to extrapolate the collision profile to large-index accuracy. It also gives a way to reweight or regenerate synthetic queries so that their collision profiles match human queries.
Core Claim
Synthetic test collections are reliable at small index sizes but become blind to geometric crowding at large ones. The reason is a measurable property: the gap between each query's relevant-document score and the upper tail of its distractor-score distribution is larger for LLM-written queries than for human queries. If this holds, two things follow. The inflated scores that Synthetic Test Collections for Retrieval Evaluation observed are not a uniform offset. They grow with index size and change system rankings among models with different robustness to scale. Matching that gap distribution, rather than prompting for "harder" queries or filtering by difficulty, is what restores agreement at scale.
For a query $q$ with relevant-document score $s^+_q$, let $F_q$ be the distribution of scores that $q$ assigns to a random non-relevant document. If distractors are treated as independent draws, the probability that the relevant document stays at rank one in an index of $m$ documents is approximately
$$P_q(\text{rank }1 \mid m) \approx F_q(s^+_q)^{\,m-1},$$
so what matters is the tail mass $1-F_q(s^+_q)$. This is the chance that a single random document outscores the answer. We estimate this tail mass by fitting a generalized Pareto distribution to the top distractor scores in a small random sample. The independence assumption is crude, since real distractors cluster by topic, as LitSearch shows. How well this approximation predicts measured large-index accuracy is itself one of the tests.
Motivation and Prior Work
Two results motivate the gap. The Curse of Dense Low-Dimensional Information Retrieval for Large Index Sizes scales MS MARCO from 10,000 to 8.8M passages. It reports a 768-dimensional DistilRoBERTa model at 88.99 MRR@10 versus 79.93 for BM25 at 10k, falling to 17.34 versus BM25's 17.56 at 8.8M. It attributes the collapse to embeddings bunching into a narrow cone ("anisotropy"), which makes collisions more likely. Synthetic Test Collections for Retrieval Evaluation generates queries and judgments with T5 and GPT-4 over MS MARCO v2 and finds a Kendall's τ of 0.8568 with human TREC DL 2023 rankings. It also finds consistently higher NDCG@10 on synthetic collections. That paper never varies index size and never asks why synthetic queries are easier. We take its collection-construction method as our strongest baseline, and its finding that sparse single-document labels give τ = 0.157 as a constraint: our synthetic collections must use dense LLM judgments, or we would be measuring a known failure.
The theory papers constrain what we should expect rather than supply our method. On the Theoretical Limitations of Embedding-Based Retrieval proves that, at a fixed dimension and a fixed required score gap, a single vector cannot separate arbitrary subsets of documents past a critical corpus size. It tests this bound on a symbolic synthetic benchmark, LIMIT. ℝ^{2k} is Theoretically Large Enough for Embedding-based Top-k Retrieval shows that, without a required gap, capacity is independent of corpus size. Once a positive score gap is required, though, capacity depends on corpus size, which places the practical bottleneck in the gap and in what models can learn. Our collision profile is an empirical, per-query version of that gap. The difference from both papers is that we measure the gap on natural queries of two different origins instead of on constructed configurations. Synthetic training methods such as Promptagator and SyNeg enter only as systems in our evaluation pool. We include them because it is plausible that retrievers trained on synthetic data are tuned to exactly the high-gap regime that synthetic evaluation rewards.
Research Program
The first stage is measurement. On MS MARCO v2 passages with TREC DL 2023 human queries and judgments, we build nested random sub-indexes of roughly 10k, 100k, 1M, 10M and the full collection. Every judged passage is always kept, so only the number of distractors changes. We generate a synthetic collection over the same passages following the recipe of Synthetic Test Collections for Retrieval Evaluation, with dense LLM judgments. The system pool is deliberately chosen to vary in robustness to scale. It includes BM25, bi-encoders we train ourselves at 128, 256 and 768 dimensions on identical data, a Promptagator-style model trained only on synthetic queries, a SyNeg-style model trained with synthetic hard negatives, and a large 4096-dimensional embedder such as GritLM. At each index size we compute system rankings under human and synthetic queries, together with each system's degradation slope, meaning its drop in NDCG@10 per tenfold growth of the index.
The second stage is the collision-profile model. For every query and system, we estimate the tail mass of distractor scores from a 10k-document random sample and predict accuracy at larger $m$ using the equation above. We then compare the distributions of tail mass for human and synthetic queries. We also check whether the difference is explained by simple surface covariates such as lexical overlap between query and source passage or query length. If it is, a cheaper fix exists.
The third stage is a margin-calibrated synthetic collection. We importance-weight synthetic queries so that their tail-mass distribution, computed under a fixed reference encoder that is excluded from the evaluated pool, matches the human distribution. We then test whether calibration transfers to systems it never saw. A second domain, LitSearch (princeton-nlp/LitSearch), provides a natural within-benchmark contrast between LLM-generated citation queries and author-written questions over 64,183 papers. There we repeat the analysis on sub-indexes down to 1k documents.
Baselines
The naive reference point is the uncorrected synthetic collection evaluated at a single small index size and treated as representative. It is the practice our hypothesis says is misleading, and its degradation slopes should be near zero for the systems that degrade most under human queries. The sparse-label variant from Synthetic Test Collections for Retrieval Evaluation, which reports τ = 0.157 on TREC DL 2023, is included only as a floor.
The strongest existing method is the fully synthetic collection with dense LLM judgments from the same paper, which reports Kendall's τ = 0.8568 against human judgments on TREC DL 2023. We rebuild it on the same MS MARCO v2 passages with the same generator and judging model and evaluate it on the identical system pool at every index size. Its τ at full scale should approximately reproduce the reported figure before any comparison is made. The margin-calibrated collection beats it if it clearly raises τ at the two largest index sizes and cuts the error in predicted degradation slope roughly in half, while not lowering τ at small sizes. With about ten systems τ is noisy, so we report bootstrap intervals over queries and count it as a win only if the intervals separate.
The difficulty-filtered baseline keeps only synthetic queries that a reference retriever fails at the small index. The hard-prompt baseline asks the LLM for ambiguous or underspecified queries. Both receive the same generation budget and the same judging pipeline as our method. The key ablation matches the synthetic collection to human queries on per-query difficulty at the small index (reciprocal rank) instead of on tail mass. This isolates whether the collision profile carries information beyond "how hard is this query today."
Why not just…?
Why not just run the synthetic collection on the full-size index? This is the obvious answer, and part of our experiment tests it directly. Running synthetic queries on the full index does include crowding. If synthetic queries have a systematically larger lead over distractors, though, they still underweight crowding, and systems that differ in robustness to scale will be ranked differently than humans rank them. The flip test decides this: the human collection ranks BM25 above the 768-dimensional model at full scale, following the pattern in The Curse of Dense Low-Dimensional Information Retrieval for Large Index Sizes, and we check whether the synthetic collection does too. If full-scale synthetic evaluation already gets the flips right, the core claim fails for the evaluation use case. What would remain is the value of cheap small-index extrapolation, which we would then report as the main result.
Why not just filter synthetic queries to the hard ones? This is a real threat. If synthetic and human queries differ only in overall difficulty, difficulty filtering is cheaper and sufficient. Our argument against it is specific: two queries can have equal reciprocal rank at 10k documents but very different tail mass, and only tail mass predicts behavior at 10M. The reciprocal-rank-matched ablation settles this. If it closes the large-index gap as well as tail-mass matching, we drop the collision profile and recommend difficulty filtering.
Why not just prompt the LLM for harder, more ambiguous queries? Prompted hardness may produce queries that are hard in the wrong way, for example vague ones that increase judging disagreement instead of crowding. The finding in Synthetic Test Collections for Retrieval Evaluation that the synthetic–human gap depends heavily on judgment design warns that changing the queries can change the labels too. We measure the tail-mass distribution of hard-prompted queries directly. If it matches human queries, prompting wins and the calibration step is unnecessary.
Why not just whiten the embeddings, or use BM25 or multi-vector models? These are fixes to retrievers, not to evaluation. The question here is whether the benchmark can see the problem at all. We include BM25 and whitened variants of our bi-encoders in the system pool because they change scale robustness, which makes the test of rank agreement harder and more informative.
Proposed Experiments and Decisive Checks
The central check has four parts, all on TREC DL 2023 over MS MARCO v2. We plot Kendall's τ between synthetic and human system rankings against index size, compare degradation slopes, and measure the tail-mass distributions of both query types. The hypothesis is supported if three things hold together. First, τ is high at 10k and drops noticeably at 10M and full scale. Second, synthetic slopes are consistently shallower for dense models but not for BM25. Third, synthetic queries have measurably smaller tail mass under every encoder in the pool.
A second check tests whether the extrapolation from the 10k sample predicts measured full-index MRR@10 per system better than simply using the 10k score. It must predict human-query and synthetic-query outcomes equally well. If it fails on clustered corpora, we replace the independence assumption with topic-stratified tail estimates.
A third check applies the reference-encoder calibration to held-out systems and to LitSearch. There, author-written queries play the role of human queries, and the reported 74.8% recall@5 for GritLM-7B on specific questions (LitSearch) is the anchor we must reproduce. No reported result found on Lacuna for any synthetic-versus-human comparison across index sizes, which is exactly the measurement this proposal supplies.
A negative result is also informative. If synthetic and human queries have the same tail mass and τ stays flat with scale, practitioners can trust small synthetic collections for scaling decisions, and the "easier collection" inflation is a uniform offset rather than a structural blind spot.
References
- Synthetic Test Collections for Retrieval Evaluation
- LitSearch: A Retrieval Benchmark for Scientific Literature Search
- The Curse of Dense Low-Dimensional Information Retrieval for Large Index Sizes
- On the Theoretical Limitations of Embedding-Based Retrieval
- $\mathbb{R}^{2k}$ is Theoretically Large Enough for Embedding-based Top-$k$ Retrieval
- Promptagator: Few-shot Dense Retrieval From 8 Examples
- SyNeg: LLM-Driven Synthetic Hard-Negatives for Dense Retrieval