When a code model scores lower on freshly written repositories than on an older benchmark, two stories fit the drop. The model may have memorized the older repositories, or the fresh repositories may simply be harder, with deeper import chains, more project-specific classes, and longer multi-file fixes. Current practice, which splits tasks by the model's training cutoff date and compares raw accuracy, cannot tell these stories apart. This proposal separates them using two things that repository code has and single-function puzzles lack. The first is a list of completion targets that provably cannot be inferred without another file. The second is a small set of measurable dependency features that let fresh tasks be reweighted to match the structure of an old benchmark before the two are compared.
Put in two sentences: if a model correctly writes the name of a class or method defined in a file it was never shown, and static analysis says that name cannot be guessed from the visible file, the model is probably recalling the repository. Once fresh tasks are reweighted so their dependency structure matches the benchmark's, any remaining score gap measures memorization rather than difficulty.
Core Claim
A score gap between a possibly contaminated repository benchmark $B$ and a pool of fresh, post-cutoff tasks $F$ can be split into a difficulty-shift part and a memorization part. The split is credible only when two conditions hold. First, the comparison is reweighted over dependency-structure features rather than prompt text. Second, a separate item-level memorization signal exists to check the split. Let $\phi(x)$ be a low-dimensional description of item $x$: the number of cross-file symbols the target needs, the import-graph distance to their definitions, the symbol kind (class, method, constant), the number of repository files, and the language. Let $s_M(x)\in\{0,1\}$ be model $M$'s exact-match success. With weights $w(x)=p_B(\phi(x))/p_F(\phi(x))$, the structure-matched expected score is
$$\hat S_F^{w}=\frac{\sum_{x\in F} w(x)\,s_M(x)}{\sum_{x\in F} w(x)},\qquad \Delta_{\text{mem}}=S_B-\hat S_F^{w},$$
where $S_B$ is the model's observed score on $B$. The estimate counts only where the fresh pool actually covers the benchmark's structure. We check this with the effective sample size $(\sum w)^2/\sum w^2$ and with per-region support, and we decline to report an estimate where coverage fails. We do not clip the weights.
The independent check is a blind-dependency probe. We take targets that the Empty Class Trick of CrossCodeEval certifies as unresolvable from the local file. That trick replaces imported classes with empty stubs and keeps only the completion points that trigger linter or compiler errors. We then show the model only the in-file context. A correct answer in this setting is either a lucky guess of a common name or recall of the repository. The guess rate is estimated from fresh items, reweighted the same way.
How This Relates to Prior Work
The closest work is Importance Sampling is All You Need (BIS). BIS predicts a model's score on a new benchmark by reweighting results from an old one. Its weights come from density models fitted to BERT embeddings of single-function docstrings, truncated at the 90th percentile. It flags contamination when the observed score departs from the prediction. It reports average absolute errors of 2.15% on pass@1 and 1.1% on code-correctness scores across HumanEval, EvoEval and BigCodeBench variants.
We borrow the reweighting idea and change three things. We weight on dependency structure instead of prompt text, because text embeddings of a cursor position say little about how far away the needed definition lives. We replace truncation with explicit coverage checks, because truncation adds bias of unknown size exactly where mismatch matters. Most importantly, we validate the contamination verdict with an item-level probe and with deliberately injected contamination. BIS's divergence check cannot separate a genuine out-of-distribution drop from memorization, and that separation is the goal here.
The sampling literature constrains the design more than it supplies it. Variational Refinement for Importance Sampling Using the Forward KL Divergence shows that proposals which fail to cover the target's tails produce unbounded weights. An approximate sampler for energy-based models with divergence diagnostics shows that reporting estimated divergences is a principled alternative to ad hoc clipping. In our setting, the fresh pool plays the role of the proposal. Both papers tell us to measure coverage over a feature space small enough to check, which is why $\phi$ is kept to a handful of interpretable variables.
On the contamination side, Automated Benchmark Generation for Repository-Level Coding Tasks (SETUPAGENT) builds SWEE-Bench and SWA-Bench. It reports that GPT-4o does better on pre-cutoff issues and that AutoCodeRover v2 falls from about 16.6% on SWE-bench to 10.2% on SWA-Bench. The same paper also reports that the new fixes are 2 to 4 times larger, so its pre/post comparison mixes both explanations. That confound is exactly what motivates this proposal.
The Emperor's New Clothes in Benchmarking? shows that surface rewrites keep fidelity above 0.9 but give almost no resistance to contamination, and that equal aggregate accuracy can hide shifts in which items a model solves. That finding motivates our item-level probe over aggregate scores. CrossCodeEval, RepoBench and RepoEval from RepoCoder avoid contamination through crawl-date windows, which age as models retrain. We treat them as the targets $B$ to be audited.
Research Program
The first component is a fresh-item generator. We re-run the CrossCodeEval extraction pipeline, including the Empty Class Trick, on repositories created after the audited model's documented cutoff. It covers Python, Java, TypeScript and C#, and it records $\phi$ for every item. It also records, for each item, whether the target is certified unresolvable.
The second component is a structural ratio estimator. It is a calibrated classifier that distinguishes $B$ items from $F$ items using $\phi$ only. We convert its outputs to $w$ and report effective sample size and per-cell support.
The third component is the blind-dependency probe, together with a consistent-rename variant. In the rename variant, the unresolvable cross-file symbol is renamed across every file of the repository so that the program still runs. The renamed definition is then supplied as context. A model that genuinely reads context will use the new name. A model that recalls the repository will tend to emit the old name.
This variant targets the point raised by the Emperor's New Clothes paper. There, surface renames failed because the model could map them back to what it had memorized. Here we score whether the model does that mapping. The old-name rate becomes a positive memorization signal rather than a leak.
The fourth component is a controlled-contamination testbed, which provides ground truth. We take an open code model whose training data predates the benchmark crawl windows, as documented by its release. We continue training it on a random half of the benchmark's repositories at several exposure counts, and we train a matched control on unrelated repositories. Contamination is then known item by item.
Baselines
The naive reference is the raw cutoff split used by SETUPAGENT: the model's score on $B$ minus its unweighted score on $F$, reported as the memorization estimate. It is run on the same items as our method. We beat it if, on the testbed, its estimate is biased in the uncontaminated control (a nonzero "memorization" that is really difficulty shift) while ours is near zero.
The strongest existing method is BIS as described in Importance Sampling is All You Need. We apply it with its IWAE density on text embeddings of the cursor-side prompt and its 90th-percentile truncation, using the same $B$, $F$ and model outputs. Its reported absolute pass@1 error of 2.15% comes from single-function benchmarks. We found no reported result on Lacuna for BIS on CrossCodeEval, RepoBench or RepoEval, so we will measure it ourselves. We give both methods the same held-out tuning set for any density-model choices. Our method beats BIS if two things hold. It must estimate the injected memorization gap with lower absolute error across exposure levels. Its predicted clean score must also have lower error on held-out fresh items.
For item-level detection, the baseline is the model's token surprise, meaning the average negative log-probability of the ground-truth completion given full context. This follows the near-zero-surprise signature noted in the contamination literature. We found no reported result on Lacuna for surprise-based detection on repository completion. It is scored by AUROC against the known contaminated/clean labels, with the same items and threshold-free evaluation.
Two ablations isolate the key ingredients. The first uses the probe computed on all targets rather than only certified-unresolvable ones; this tests whether the unresolvability filter matters. The second uses our estimator with text-embedding weights in place of structural weights; this tests whether the features matter.
Why not just…?
Why not just compare pre-cutoff and post-cutoff scores? SETUPAGENT's own finding that post-cutoff fixes are 2 to 4 times larger shows that the two pools differ in difficulty, so the raw gap overstates memorization. This remains a real threat in one case. If the dependency features of $B$ and $F$ turn out nearly identical, the weights will be near 1 and our method will reduce to this baseline. The uncontaminated-control run settles the question: a raw gap well above zero in that run is direct evidence of confounding.
Why not just use token surprise on the answer? This is the most serious alternative because it is cheap and needs no fresh pool. Its weakness is that surprise is also low for easy, locally predictable targets, so it confuses easy items with recalled ones. The probe restricts attention to targets that static analysis says cannot be predicted locally. In effect, the probe is surprise restricted to targets that the visible file cannot explain. If unrestricted surprise matches the probe's AUROC on the testbed, we will drop the probe and keep only the structural reweighting for the aggregate estimate. The unresolvability ablation decides this directly.
Why not just apply BIS as published? BIS weights on docstring embeddings. For repository completion, the thing that sets difficulty is the distance and number of cross-file definitions, and prompt text hardly encodes it. RepoGraph supports this view: it improves localization from 47.3% to 51.7% by exploiting exactly this graph structure. BIS also truncates weights instead of checking coverage. The head-to-head comparison and the text-versus-structure ablation measure both differences.
Why not just keep crawling fresh repositories? That is what builds $F$, so we do not reject it. But freshness alone only gives a clean score, not the answer to how much of an older benchmark's score was recall. It also expires as new models are trained.
Proposed Experiments and Decisive Checks
The primary benchmarks are CrossCodeEval and RepoBench-C/P as $B$, with RepoEval as a third target, using exact match and edit similarity as the benchmarks define them. SWEE-Bench issues split by creation date serve as a secondary, agent-level setting. There, only the reweighting applies, with fix size and file count used as features.
The first decisive check is null calibration. On the control model, both $\Delta_{\text{mem}}$ and the probe's excess over the reweighted guess rate must be statistically indistinguishable from zero. If they are not, the method fails.
The second check is dose response. As exposure count and the contaminated fraction rise, $\Delta_{\text{mem}}$ should increase monotonically and track the true gap on the injected half. Within the contaminated model, the probe success rate should separate injected repositories from held-out ones.
The third check is abstention honesty. In feature regions where the effective sample size is small, for example deep import chains in rarely covered languages, the estimator must decline to report a value. We will verify that forcing an estimate in those regions produces large errors, which shows that the coverage check is doing real work.
Only after these checks pass will we audit public models on CrossCodeEval and RepoBench. There we will report, per model, the structure-matched memorization estimate and the probe and rename signals. We will note one caveat: the probe detects exposure to the repository itself, which for crawl-based completion benchmarks is exactly the contamination in question.
References
- CrossCodeEval: A Diverse and Multilingual Benchmark for Cross-File Code Completion
- Importance Sampling is All You Need: Predict LLM's performance on new benchmark by reusing existing benchmark
- Variational Refinement for Importance SamplingUsing the Forward Kullback-Leibler Divergence
- An approximate sampler for energy-based models with divergence diagnostics
- Automated Benchmark Generation for Repository-Level Coding Tasks
- The Emperor's New Clothes in Benchmarking? A Rigorous Examination of Mitigation Strategies for LLM Benchmark Data Contamination
- RepoBench: Benchmarking Repository-Level Code Auto-Completion Systems
- RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generation
- RepoGraph: Enhancing AI Software Engineering with Repository-level Code Graph