Did the Backbone See the Test Set? Contamination Audits for Deep Hashing Benchmarks

Image retrieval systems that search millions of pictures usually compress each image into a short binary code, a string of 16 to 64 bits, so that comparing two images costs a single bitwise operation. This family of methods is called deep hashing. The best recent methods no longer learn these codes from raw pixels. They start from a large pre-trained vision-language model such as CLIP, which was trained on hundreds of millions of web images. They are then scored on old academic benchmarks such as CIFAR-10, MSCOCO, NUS-WIDE and Flickr25k, and those benchmark images have circulated on the web for over a decade. When a method reports 0.940 mean average precision on CIFAR-10, nobody can currently say how much of that comes from the hashing algorithm and how much comes from a backbone that may already have seen, and organized, those very images.

We propose a measurement protocol that answers this question. It splits a hashing method's reported score into three parts: what the frozen backbone gives for free, what the method adds, and how much of either part disappears on freshly collected images that the backbone cannot have seen. The difficulty is that fresh images also differ from old ones in ordinary ways. To separate memorization from that ordinary shift, we compare against a backbone that is known never to have seen the benchmark. We calibrate the whole protocol with deliberately contaminated backbones, so that the size of the score drop it reports can be read as a dose of contamination rather than guessed.

Core Claim

The headline claims of CLIP-backed unsupervised hashing are currently unfalsifiable. The clearest example is the claim that external text guidance breaks an "intrinsic ceiling" of visual-only hashing. Deep Unsupervised Hashing via External Guidance (DUH-EG) uses CLIP to match unlabeled images to clustered WordNet nouns and uses those matches to remove false negatives during contrastive training. It reports 0.940 mean average precision on CIFAR-10, against roughly 0.88 for the earlier methods DDCH and HHCH. Separately, Both Text and Images Leaked! shows that multimodal models absorb standard benchmarks, MSCOCO among them, and that the leakage can begin as early as unimodal pre-training. These two findings do not contradict each other, but together they mean the 6-point gain could come from a better algorithm, from the backbone's memorized arrangement of benchmark images, or from both.

We claim this can be settled with a difference-in-differences design, which compares how much a score falls on new data for a model that may have seen the benchmark against how much it falls for a model that certainly has not. The score drop from an original benchmark to a fresh "twin" benchmark has two causes. One is ordinary shift, which would affect any model. The other is memorization, which affects only a backbone that saw the benchmark. A backbone guaranteed clean of the benchmark experiences only the first cause, so subtracting its drop isolates the second. What is new is not the use of fresh data, which Clean Evaluations on Contaminated Visual Language Models and Unveiling Causal Reasoning in Large Language Models: Reality or Mirage? already use. What is new is this subtraction, the calibration against known doses of contamination, and applying both to ranked nearest-neighbor retrieval in Hamming space (the space of binary codes, where distance is the number of differing bits) rather than to question answering.

Research Program

The first component is a set of fresh twin benchmarks. For CIFAR-10 we collect images of the same ten classes that were published after the training cutoff of the backbones under test. We downsample them to 32×32 and filter them to match the original on color statistics and on the distribution of class-confidence scores from a weak classifier. For MSCOCO, NUS-WIDE and Flickr25k we collect post-cutoff multi-label images that match the original label co-occurrence statistics. We follow the temporal-slicing logic of the Gamersky benchmark in Clean Evaluations on Contaminated Visual Language Models, which used images published within a fixed post-cutoff week. We then remove any twin image that is a near-copy of an existing web or benchmark image. For this we use the compact copy-detection descriptors of Relational Self-Supervised Distillation with Compact Descriptors for Image Copy Detection (RDCD), which reports µAPsn of 61.1 on DISC2021 with 128-dimensional descriptors. That descriptor size is small enough to run against large corpora. The goal is that the twins share the benchmarks' categories but none of their pixels.

The second component is a set of calibrated contaminated backbones. Retraining CLIP from scratch is out of reach, so we adapt the clean-versus-contaminated design of Clean Evaluations on Contaminated Visual Language Models. We start from an open CLIP-style backbone and run continued contrastive image-text training on a mid-sized corpus. First we purge that corpus of near-copies of every benchmark, again using RDCD descriptors. From this purged corpus we produce a clean backbone and a series of contaminated backbones. Each contaminated backbone has the benchmark's gallery and query images, with their captions or class names, injected at a different dose: zero, one, and several passes. Every hashing method is then trained on top of each backbone. This yields dose-response curves showing how much CIFAR-10 or MSCOCO mean average precision inflates per unit of contamination. Those curves let us convert the drop observed for the original public CLIP into an estimated equivalent dose.

The third component tests whether memorization matters more at scale. Retrieval benchmarks use small galleries, but deployed systems search millions of items. We add up to millions of fresh, near-copy-purged distractor images to the gallery. We then ask whether memorized backbones keep benchmark images clustered tightly enough to resist those distractors, which would make the contamination gap grow with gallery size. The alternative is that the gap stays flat.

Formally, for a method $m$ on backbone $b$, let $A_{\text{orig}}(m,b)$ and $A_{\text{twin}}(m,b)$ be mean average precision on the original benchmark and on its twin. Let $b_0$ denote the purged clean backbone. The memorization share is then

$$C(m,b) = \big[A_{\text{orig}}(m,b) - A_{\text{twin}}(m,b)\big] - \big[A_{\text{orig}}(m,b_0) - A_{\text{twin}}(m,b_0)\big].$$

The quantity we care most about is $C$ for DUH-EG minus $C$ for frozen-backbone hashing. This difference measures how much of DUH-EG's gain over frozen-backbone hashing relies on memorization.

Baselines

The naive reference point is frozen-backbone hashing. We take the frozen CLIP image embedding and binarize it, either with random-projection locality-sensitive hashing (random hyperplanes followed by taking the sign) or by taking the sign after a principal-component rotation. It involves no learning beyond the backbone, so whatever it scores is a free gift from pre-training. It is the most important baseline: if it already approaches 0.940 on CIFAR-10, the hashing literature's gains are small on top of the backbone, and memorization concerns attach mostly to the backbone. We found no reported result on Lacuna for frozen-CLIP sign hashing on CIFAR-10. We will run it at 16, 32 and 64 bits on the same splits as DUH-EG.

The strongest existing method is DUH-EG, which reports 0.940 mean average precision on CIFAR-10 at 16, 32 and 64 bits with a ViT backbone, per Deep Unsupervised Hashing via External Guidance. We also run DDCH and HHCH, reported there at about 0.88. To place the pre-foundation-model regime, we cite Shadow Recurrent Hashing, which A survey on deep hashing for image retrieval reports at 0.6166 at 12 bits. Every method is run on every backbone, original and purged, with identical data splits, code lengths and hyperparameter search budgets. DUH-EG's WordNet anchor construction is rerun for each backbone rather than cached.

The ablation that isolates the proposal's key ingredient is the clean-backbone subtraction. We report the naive twin drop $A_{\text{orig}} - A_{\text{twin}}$ alongside the corrected $C$, and on the calibration backbones we check which of the two tracks the injected dose. A second ablation removes DUH-EG's external text guidance, testing directly whether the guidance is what carries the memorized component.

We will count our protocol as beating the naive twin comparison if two things hold. First, $C$ must be monotone in injected dose and near zero for the clean backbone. Second, the naive drop must fail at least one of those two properties, for instance by staying large even on the clean backbone because the twins are simply harder.

Why not just…?

Why not just evaluate on fresh post-cutoff images and call any drop contamination? This is the most serious alternative, and it is the uncorrected arm of our ablation. Its weakness is that fresh images differ in camera hardware, style and composition, so every model drops on them. The large declines reported on CausalProbe-2024 in Unveiling Causal Reasoning in Large Language Models: Reality or Mirage? are exactly the kind of result where memorization and shift cannot be separated without a control. If our clean backbone shows the same drop as the public one, then the simple approach was right about the size of the drop but wrong to attribute it to contamination. We will report that outcome.

Why not just deduplicate the benchmark against the pre-training corpus? We do this wherever we control the corpus. It is how the clean backbone is built. For proprietary backbones the corpus is unavailable, and Both Text and Images Leaked! notes that exact overlap quantification across private pre-training data is impractical. Deduplication also catches only copies. It cannot detect a backbone that has learned the benchmark's distribution through re-encoded or rephrased duplicates. The calibration backbones test how much inflation survives deduplication thresholds at different strictness levels.

Why not just use the color-channel-swap "poison pill" from VLM contamination work? Clean Evaluations on Contaminated Visual Language Models found that swapping red and blue channels removed memorization gains while staying ungameable, because training on swapped images damaged the model (InternVL BLEU 0.1041 against a clean 0.1047). Hashing, however, compresses features into a few bits. A channel swap may scramble Hamming neighborhoods for clean and contaminated backbones alike, which would make it useless as a diagnostic. We include the swap as a candidate test on the calibration backbones. It counts as a cheap substitute for twin collection only if its drop separates the dose levels while leaving the clean backbone nearly intact.

Why not just hide the gallery behind a private evaluation server? A private server prevents future test-set leakage. It does nothing about images already absorbed during backbone pre-training, which is the question here. It also turns evaluation into a black box, a problem $P^4$Bench raises against private benchmarks.

Proposed Experiments

The first decisive check is the calibration. On the purged backbone, injecting CIFAR-10 and MSCOCO test images must inflate original-benchmark mean average precision in a dose-dependent way, and $C$ must recover that ordering. If the injection produces no inflation, then memorization is not a mechanism worth auditing in this setting, and the project ends with that negative result.

The second check is the main audit. We estimate $C$ for frozen-backbone hashing, DDCH, HHCH and DUH-EG on the public CLIP backbone across CIFAR-10, MSCOCO, NUS-WIDE and Flickr25k at 16, 32 and 64 bits. We report mean average precision and precision at fixed Hamming radius, with confidence intervals from bootstrapping over twin images. The ceiling-breaking claim survives if DUH-EG's advantage over frozen-backbone hashing persists on the twins and on the purged backbone. It is undermined if the advantage concentrates in $C$.

The third check is the gallery-scaling experiment, which plots $C$ against distractor count from thousands to millions of images. It tests whether memorized geometry protects benchmark neighbors more as the search space grows. As an exploratory extension, the same twin-and-control design applies to geolocation retrieval with GeoCLIP on Im2GPS3k, where GeoCLIP reports 14.11% street-level accuracy. We would treat it as secondary because building a matched geolocation twin is considerably harder.