Are Adapter Gains on Compositional Benchmarks Real or Memorized?

A common way to improve a large pretrained model cheaply is to freeze it and train a small add-on module, a "plug-and-play adapter," often using hard negatives. Hard negatives are distractors that match the correct answer almost exactly but differ on one detail, such as "a red cup on a blue table" versus "a blue cup on a red table." Gains are then reported on benchmarks built from MS-COCO, a photo dataset that has circulated on the web for a decade. This practice rests on an assumption nobody tests. It assumes the frozen backbone has never seen those benchmark images and captions, so any improvement reflects better understanding of how objects, attributes and relations combine. If the backbone has memorized COCO image–caption pairs during web-scale pretraining, an adapter trained with hard negatives may learn to sharpen that memorized association rather than learn composition.

We propose a controlled measurement that separates the two. We pair each COCO-derived test item with a matched "fresh twin" built by the same pipeline on images the backbone cannot have seen. We then ask whether an adapter's gain concentrates on the items the backbone has likely memorized. In two sentences: benchmark gains from hard-negative adapters on frozen vision-language models should be measured as a difference-in-differences between contaminated and fresh test items. We build the fresh items, a per-item memorization score, and the injection experiment needed to validate it, and we use them to find out which hard-negative training recipes survive.

Core Claim

The claim is testable. For a frozen dual encoder such as CLIP (an image model and a text model trained so matching image–caption pairs have high similarity), part of the improvement that hard-negative adapters show on SugarCrepe++ comes from items whose source image–caption pair the backbone memorized. That part will vanish on matched fresh items. The share should be largest when the adapter's hard negatives are themselves mined from COCO, and smallest when they come from disjoint images using across-scene swaps. The paper is valuable whichever way the result falls. A near-zero contamination share would vindicate the frozen-backbone assumption for this setting. A large share would mean the field's compositionality leaderboard partly measures memorization.

Why This Is Not Already Done

Three findings motivate the question. SugarCrepe showed that earlier compositionality benchmarks could be solved by text-only models that simply pick the fluent sentence. It also showed that the gains of NegCLIP, which was trained on rule-based hard negatives, vanished on clean data. So hard-negative training can latch onto shortcuts invisible to the benchmark. SugarCrepe++ added a second, lexically different correct caption. It showed that CLIP-style models fall to 45–55% on swap tasks and attributed this to lexical bias. It never asked whether sensitivity to wording partly reflects memorization of the benchmark's source captions. Dong et al. showed that contaminated language models produce abnormally "peaked" outputs, and they detect leaks from this. Their method measures edit distance across sampled text, which has no analogue for a dual encoder that outputs a single similarity score.

The closest work is CLIC. CLIC excludes MS-COCO from training to avoid "knowledge leakage." It builds hard negatives by concatenating two unrelated images and swapping words across their captions, which rules out false negatives, and it reports a +9% average gain on SugarCrepe++ image-to-text with CLIPS. CLIC controls leakage through its fine-tuning data but not through the backbone's pretraining. It also updates the encoders rather than freezing them. UniME supplies the adapter-plus-hard-negatives recipe: QLoRA on a frozen multimodal LLM, with a similarity threshold that discards in-batch negatives identical to the positive. Its "false negative contamination" concerns collisions within a training batch, not test data leaking into pretraining.

On the plug-and-play side, detection adapters such as Fan's mixture-of-experts router explicitly rely on frozen DINOv2 and DETR remaining trustworthy. We use them only as a secondary check. No paper in our search puts backbone memorization, adapter training and hard-negative evaluation into one controlled design.

Research Program

The first component is a fresh-twin benchmark. We collect about 5,000 photographs taken after the release dates of the backbones under study, with scene statistics roughly matched to COCO. Each gets human-written captions. We then run the SugarCrepe++ construction on them: generate two paraphrased positives and one hard negative in each of the five categories (swap object, swap attribute, replace object, replace attribute, replace relation). Humans verify the items. Following SugarCrepe, we prune items adversarially until text-only models sit at chance, so the fresh items cannot be solved by fluency cues.

A known risk, flagged by the synthetic-benchmark literature including DyVal, is that generated items carry their own regularities. We therefore apply the identical generator and filter to a re-generated COCO split. This way, COCO and fresh items differ only in image provenance, not in how the negatives were written.

The second component is a per-item memorization score for dual encoders. It is the counterpart of peakedness for models that output one score per pair. For a COCO image, we compare the backbone's similarity to its original COCO caption with its similarity to several human-verified paraphrases that carry the same meaning. We standardize this excess against the same statistic on fresh images. A memorized pair should show an original-caption advantage that meaning alone cannot explain.

We validate the score before trusting it. Following the controlled-exposure logic of the benchmark watermarking study, we continue pretraining an OpenCLIP model on web data. Into that data we inject subsets of fresh-twin pairs at known repetition counts, including zero. We then check whether the score ranks injected pairs above non-injected ones and whether it grows with exposure.

The third component is the adapter study. Every adapter is a low-rank module on a frozen backbone, trained with an identical parameter budget and training-step budget. Only the hard-negative source varies. One source is COCO-derived rule-based negatives, the NegCLIP style. A second is COCO-free within-scene LLM swaps. A third is COCO-free across-scene swaps in the CLIC style, with UniME-style false-negative thresholding. The primary quantity is

$$\Delta = \big(A_{\text{adapt}} - A_{\text{base}}\big)_{\text{COCO}} - \big(A_{\text{adapt}} - A_{\text{base}}\big)_{\text{fresh}},$$

where $A$ is the SugarCrepe++ 3-way hit rate (both positives must beat the negative). The subscripts denote the adapted or frozen base model and the test split. Distribution shift between splits moves both the base model and the adapter, so it largely cancels. $\Delta$ is the part of the gain that exists only on COCO. Within COCO, we also regress per-item gain on the memorization score.

Baselines

The naive reference point is the frozen backbone with no adapter, evaluated on both splits. Its COCO-minus-fresh gap alone tells us how much the backbone benefits from familiarity. Against it, even a well-behaved adapter should show a positive gain on both splits. We use OpenCLIP and the CLIPS model in addition to OpenAI CLIP, so that at least one backbone has a documented pretraining corpus where COCO overlap can be checked directly.

The strongest existing method is CLIC (CLIC). It reports a +9% average improvement on SugarCrepe++ image-to-text when applied to CLIPS. We run its full fine-tuning in two forms: the released recipe as published, and a low-rank adapter version trained on the same data, the same number of steps and the same hyperparameter search budget as our adapters. This lets us separate the effect of freezing from the effect of the negatives.

We also include a UniME-style adapter with thresholded in-batch hard negatives (UniME). UniME reports 70.7% on MMEB versus 63.3% for VLM2Vec. On SugarCrepe, no reported result was found in our search for UniME.

The key ablation holds the architecture, budget and loss fixed and swaps only the negative source between COCO-derived and COCO-free. This isolates whether mining negatives from benchmark-adjacent data is what inflates $\Delta$.

Beating CLIC here does not mean a higher COCO score. It means matching CLIC's SugarCrepe++ gain on the fresh split while showing a $\Delta$ whose confidence interval covers zero. We compute that interval by bootstrap over items, with three training seeds per adapter.

Why not just…?

Why not just exclude COCO from adapter training, as CLIC already does? This removes leakage from the training data, but the backbone was pretrained on web crawls that plausibly contain COCO images and captions. The concern is that the adapter amplifies memorization already inside the frozen model. The COCO-free arms of our study test this directly. If their $\Delta$ is near zero, the reviewer is right and the simple fix suffices. We would report that as a finding.

Why not just run a text-only "blind" baseline, as SugarCrepe did? Blind models catch artifacts in how the negatives are written. They cannot detect memorization of image–caption pairs, because memorization requires the image. Our adversarial filter already drives blind accuracy on both splits to chance. Any remaining COCO-specific advantage therefore cannot be a text artifact.

Why not just evaluate on a new held-out benchmark and skip the difference-in-differences? A raw drop on new images confounds contamination with ordinary distribution shift and difficulty. That is exactly the ambiguity in reports of performance collapsing on post-cutoff data such as Chi et al.. Subtracting the base model's own drop removes shift that affects both models. The within-COCO memorization regression gives a second, split-free line of evidence.

Why not just use an existing contamination detector? CDD and TED from Dong et al. need many sampled text outputs per prompt, which a dual encoder does not produce. Watermarking only works for benchmarks marked before release, and COCO was not. The paraphrase-excess score is our substitute, and the injection experiment decides whether it works. If it fails to separate injected from non-injected pairs, we drop the per-item analysis and rely on the split-level $\Delta$ alone.

Proposed Experiments and Decisive Checks

We start with validation. The memorization score should detect injected fresh pairs above chance, with ROC AUC rising monotonically in exposure count. The +5% inflation threshold at which watermarking reached $p=10^{-3}$ on ARC-Easy gives a sensitivity target to aim for in the vision setting, though the settings are not directly comparable.

Next, we measure the frozen baselines' COCO-minus-fresh gap on SugarCrepe++ categories. We expect swap categories to sit near the reported 45–55% on COCO. If fresh items score at that level too, backbone memorization is not driving swap failures.

The central experiment computes $\Delta$ for all adapter arms, across three backbones and three seeds. The hypothesis is falsified if every arm, including COCO-derived negatives, shows $\Delta$ indistinguishable from zero and no correlation between per-item gain and memorization score. It is supported if the COCO-negative arm shows significantly positive $\Delta$ and the across-scene arm does not.

As a secondary check on the frozen-backbone assumption in detection, we apply the same logic to Fan's router. That method reports 39.8 mAP at 10-shot on the six-domain cross-domain few-shot detection benchmark, against 29.6 for CD-ViTO. For each target domain, we compare its gain over frozen DINO-DETR on the benchmark's public images with its gain on a small set of newly photographed images of the same object categories. This extension is exploratory. Its sample sizes will be small, so we will report it as such.