Generative image upscalers are now trained on web-scale photo collections and then scored on a handful of public benchmark images: DIV2K, Urban100, Set5, UH4K. These models invent fine detail that the low-resolution input does not contain. When such a model reproduces a benchmark's true fine texture, we cannot currently tell whether it inferred that texture or had seen the image before. Language-model research has shown that even one leaked copy of a test set can push scores past what clean scaling allows, and that leaked models behave unlike clean ones when their sampling is randomized. Nobody has measured the image analogue, and the usual evaluation hides it. Full-reference scores such as PSNR (peak signal-to-noise ratio) are already so badly matched to generative models that a foundation model can trail simple regressors by more than 4 dB on DIV2K while looking better.
We propose a black-box leakage test for generative super-resolution. The test needs no access to the training data. It rests on one observation. Detail above the input's sampling limit, the Nyquist frequency, is information the input does not carry. A model that has never seen the ground truth can produce such detail only as a guess. Its samples should therefore agree with the hidden ground truth no more than they agree with each other, once this is calibrated on images the model provably never saw. A leaked image breaks this pattern. Different random seeds each land closer to the true image than to one another, because each seed is the memorized image plus independent sampling noise. We will validate the test by deliberately injecting benchmark images into the training data at controlled doses. The same experiments will measure how much leakage inflates benchmark scores as resolution grows.
Core Claim
Write $x$ for the ground-truth high-resolution image and $y_1,\dots,y_S$ for $S$ outputs of a diffusion upscaler (a generator that turns noise into an image by repeated denoising, here conditioned on the low-resolution input) run with different random seeds. Let $P_B$ keep only the spatial-frequency band $B$ above the input's Nyquist limit, and let $\rho$ be Pearson correlation computed patch by patch. We define the beyond-Nyquist excess agreement as
$$\Delta_B(x) = \frac{1}{S}\sum_s \rho(P_B y_s, P_B x) \;-\; \frac{2}{S(S-1)}\sum_{s<s'} \rho(P_B y_s, P_B y_{s'}).$$
The first term measures how well samples match the truth in the band. The second measures how well samples match each other. For a sampler that is calibrated and has no knowledge of $x$ beyond the input, the truth is exchangeable with a fresh sample, so the expected value of $\Delta_B$ is zero. Real models are miscalibrated, sometimes overconfident and sometimes overdispersed, so we do not test against zero. Instead we estimate each model's own null distribution of $\Delta_B$ on freshly captured, never-published high-resolution photos. We then flag benchmark images, or individual patches, whose $\Delta_B$ is an outlier on the positive side.
The claim is that this statistic detects benchmark leakage at doses as low as one training copy, where score inflation is invisible in PSNR or LPIPS (a learned perceptual distance). It also localizes leakage to patches. Patch-level localization matters because upscalers are trained on random crops, and convolutional diffusion models build outputs as mosaics of locally matched training patches (An analytic theory of creativity in convolutional diffusion models). Leakage in this setting is therefore naturally partial and sub-image.
Grounding in Prior Work
Two papers motivate the study from the contamination side. Quantifying the Effect of Test Set Contamination on Generative Evaluations and Causally Quantifying the Effect of Test Set Contamination on Generative Benchmarks inject 0 to 3,162 copies of the MATH test set into a pretraining corpus. Even one copy pushes test loss below the clean "irreducible" floor. At high temperature, accuracy of contaminated models collapses up to 40x faster than that of clean models. Both papers are restricted to text language models. Our seed-based statistic is the continuous analogue of their temperature diagnostic. It replaces "does accuracy collapse under randomness" with "does randomness scatter samples around the truth rather than around each other." Soft Contamination Means Benchmarks Test Shallow Generalization shows that near-duplicates evade exact filters, for example in 78% of CodeForces problems in the Olmo3 corpus. That finding motivates our near-duplicate injection arm: re-encoded, rescaled, and re-cropped copies.
Three papers supply mechanism and method on the image side. Local Coverage Governs Memorization in Diffusion Models shows that diffusion models memorize isolated points in sparse regions while generalizing in dense ones. Its experiments use only 32×32 images in pixel space. That paper predicts where leakage should bite. Distinctive benchmark textures should be more exposed than generic sky or foliage. We test this prediction at 2K and 4K and in a compressed latent space. Could It Be Generated? introduces InvMM, which inverts an image into the noise space of a text-to-image model. It detects memorized training members with AUC 0.997 on CIFAR-10 and 0.974 on LAION. InvMM is our strongest existing competitor. It was built for unconditional and text-conditioned generation and for privacy, not for conditional upscalers or benchmark auditing.
One paper is the reason the problem matters in practice. Is Nano Banana Pro a Low-Level Vision All-Rounder? evaluates a web-trained foundation model on 40 low-level datasets. It never asks whether those datasets were in pretraining. Its large PSNR deficit also shows that full-reference scores cannot be the leakage signal.
Research Program
The core of the program is a controlled injection study for super-resolution. We train two families of upscalers. One is an SRDiff-style pixel-space residual diffusion model with about 12M parameters (SRDiff). The other is a latent-diffusion upscaler that operates on compressed codes. Both are trained on LSDIR (The Tenth NTIRE 2025 Efficient Super-Resolution Challenge Report), with all DIV2K and Urban100 images and their near-duplicates removed by perceptual-hash and patch nearest-neighbor filtering.
Into this clean corpus we inject a random half of the DIV2K validation images and of Urban100 at doses of 0, 1, 3, 10, 32, and 100 copies. The other half stays clean as an internal control. A second arm injects only a fraction of each image's crops, from 10% to 100% spatial coverage, to test patch-level leakage. A third arm injects degraded near-duplicates: JPEG re-encoding, 0.9× rescaling, and shifted crops.
We cross these arms with training crop size and test resolution (DIV2K at 2K versus UH4K at 4K). This asks whether leakage inflation grows or shrinks with resolution. A single 4K image yields many more training crops per copy. Latent compression, in contrast, might wash out the fine detail that would otherwise be memorized. Which effect dominates is the open question raised by the pixel-space-only analysis in Local Coverage Governs Memorization in Diffusion Models.
On each trained model we compute $\Delta_B$ maps. The null is estimated on a fresh set of about 300 high-resolution photos captured for this project, matched to DIV2K scene categories and never published until the study ends. We report image-level and patch-level detection performance as a function of dose. Finally, we apply the calibrated test to publicly released web-trained diffusion upscalers on the LowLevelEval suite (jlongzuo/LowLevelEval). This last step is an audit of real models, not a validation, and we will report it with that caveat.
Baselines
The naive reference is score inflation itself: the PSNR and LPIPS gap between a contaminated model and its clean twin on the injected half of the benchmark. It requires the clean twin, so it is unavailable in real audits. In the injection study it tells us how large the inflation is that we are trying to detect. We do not have a Lacuna-reported PSNR for SRDiff-style models on DIV2K. The only reported anchors are the NTIRE 2025 qualification bar of PSNR at or above 26.90 dB on DIV2K_LSDIR_valid and the more than 4 dB specialist gap reported for Nano Banana Pro. We will therefore report our own clean-model numbers.
The strongest existing method is InvMM (Could It Be Generated?). Its reported numbers are AUC 0.997 on CIFAR-10 and 0.974 on LAION, both for training-member detection, not for benchmark leakage in upscalers. We adapt it by inverting the high-resolution target into the upscaler's initial noise with the low-resolution input held fixed, using the paper's AIMD weight schedule. We give it the same per-image compute budget as our method: wall-clock time matched to $S$ forward samples. Beating InvMM means higher image-level AUC at doses of 1 and 3 copies, where inflation is smallest, and a lower false-positive rate on the held-out fresh photos.
A second strong competitor is the fresh-set comparison familiar from live benchmarks (LiveBench). It compares a model's mean score on the benchmark with its mean score on newly captured images. We run it on exactly the same fresh set. It gives only a benchmark-level verdict. Beating it means correct per-image and per-patch flags, and correct verdicts when the fresh set is deliberately shifted in difficulty, for example with more texture-heavy scenes.
An oracle baseline is patch nearest-neighbor search against the training corpus. It is available only because we built the corpus, and it serves as an upper bound on what any black-box test could achieve.
Three ablations isolate the key ingredients. Seed agreement alone is the direct temperature analogue: it asks whether samples become suspiciously consistent. The match term alone, sample-to-truth agreement without subtracting sample-to-sample agreement, tests whether the subtraction matters. Full-band computation in place of beyond-Nyquist computation tests whether restricting to frequencies the input cannot explain is essential.
Why not just…?
Why not just search the training data for the benchmark images? When the corpus is open, this works, and we use it as the oracle. The real cases are opaque web-scale corpora, which is why Is Nano Banana Pro a Low-Level Vision All-Rounder? could not audit its model. The relevant question is how close a black-box test gets to the oracle, and our dose-response curves measure exactly that gap.
Why not just compare scores on a fresh, never-published test set? This is a genuine threat. If benchmark leakage only ever happened wholesale, a matched fresh set would detect it more cheaply. It has two weaknesses. Any difficulty mismatch between fresh and benchmark images confounds the score gap. It also cannot localize partial, patch-level leakage. Our method still uses fresh images, but only to estimate the null distribution of a statistic normalized by sample-to-sample agreement, and that normalization should make it less sensitive to difficulty. The shifted-difficulty experiment settles the matter. If the fresh-set comparison is equally robust there and equally sensitive at one copy, we will report that the simpler test suffices for image-level audits.
Why not just apply InvMM unchanged? InvMM answers whether some noise vector reproduces the image. For a conditional upscaler, a good model can reproduce much of any image from its low-resolution input, so the question is less discriminative. InvMM also costs an optimization per image. Head-to-head AUC under matched compute will show whether its reported 0.974 on LAION carries over to this setting.
Why not just measure how consistent the seeds are, by analogy to the temperature test? Clean generative models are often overconfident, and their samples agree with each other on regular structures, so consistency alone would flag clean outputs on self-similar Urban100 facades. The seed-agreement-only ablation tests this directly. We expect a high false-positive rate on Urban100 and a much lower one for $\Delta_B$.
Decisive Checks
The proposal succeeds under four conditions. First, $\Delta_B$ separates injected from held-out benchmark images at a dose of one copy with AUC clearly above InvMM and the seed-agreement ablation, while the PSNR gain at that dose remains small. Second, patch maps from the partial-crop arm recover which regions were leaked, scored against the known injection masks. Third, false-positive rates on clean Urban100 stay at the level estimated from the fresh-photo null, showing that natural self-similarity is not mistaken for memorization. Fourth, the resolution and latent-space comparison shows a measurable direction: whether leakage inflation per copy grows from 2K to 4K, and whether latent compression suppresses it. That last result is useful either way. It tells practitioners whether high-resolution benchmarks are more or less fragile than the low-resolution cases studied so far.
The proposal fails if $\Delta_B$ is no better than the fresh-set mean comparison under difficulty shift. It also fails if the miscalibration of clean models varies so much across images that no per-model null is stable. We would report either outcome as a negative result.
References
- An analytic theory of creativity in convolutional diffusion models
- Quantifying the Effect of Test Set Contamination on Generative Evaluations
- Causally Quantifying the Effect of Test Set Contamination on Generative Benchmarks
- Soft Contamination Means Benchmarks Test Shallow Generalization
- Local Coverage Governs Memorization in Diffusion Models
- An Inversion-based Measure of Memorization for Diffusion Models
- Is Nano Banana Pro a Low-Level Vision All-Rounder? A Comprehensive Evaluation on 14 Tasks and 40 Datasets
- SRDiff: Single Image Super-Resolution with Diffusion Probabilistic Models
- The Tenth NTIRE 2025 Efficient Super-Resolution Challenge Report
- LiveBench: A Challenging, Contamination-Limited LLM Benchmark