Poisoning the Frozen Backbone: Cheap Availability Protection Against Adapter-Based Scraping

Creators who publish images online face a simple problem: anyone can scrape their work and train a model on it without permission. One proposed counter is "unlearnable examples" — adding invisible noise to images before publication so that any model trained on them learns nothing useful, while humans still see the original picture. The catch is that this whole research line assumes the scraper trains a network from scratch. Today, almost nobody does that. A scraper instead downloads a large pretrained vision model (for example DINOv2 or OpenCLIP), freezes it, and bolts on a small trainable piece — a linear head, an adapter, or a handful of prompt tokens. This costs hours, not thousands of GPU hours. There is good evidence that pretraining already defeats standard protections: fine-tuning a pretrained ResNet-18 or Tiny-ViT on standard unlearnable data recovers 61.82% and 81.40% accuracy respectively, versus near-chance for training from scratch (When Priors Backfire: On the Vulnerability of Unlearnable Examples to Pretraining). But no one has measured whether the cheapest adaptation of all — a frozen backbone plus a small trainable module — also walks straight past these protections; the closest work, Robustness Tokens: Towards Adversarial Robustness of Transformers, uses that setup only as a test-time defense and never on poisoned training data.

This proposal has two parts. First, a measurement study: quantify how cheaply an adapter-based scraper bypasses each major protection family. Second, a defense built for that threat model: instead of poisoning the scraper's private training dynamics (which requires expensive bi- or tri-level optimization, e.g. over 50 GPU hours for Robust Unlearnable Examples: Protecting Data Privacy Against Adversarial Learning), poison the one thing that is public and shared — the frozen backbone's representation. Because the defender knows the exact feature map the scraper will reuse, the optimization collapses to a single level and becomes roughly as cheap as generating one adversarial example per image. The proposal treats unauthorized data exploitation as the core problem, adaptive attack vulnerability as the evaluation discipline, computational overhead as the axis on which we claim a win for the defender, and the robustness–accuracy trade-off as the recurring lens (defenses that scrub shortcuts tend to hurt clean accuracy; cheap adaptation tends to sacrifice resilience).

Core Claim

Availability protection should target the frozen, publicly downloadable representation, not the from-scratch training loop. Concretely: perturb each image so that its embedding under a small ensemble of public backbones collapses onto a class-agnostic anchor, making the protected data uninformative for any head, adapter, or prompt module trained on top — at a small fraction of the generation cost of Robust Unlearnable Examples, Provably Unlearnable Examples (45 GPU hours on ImageNet), or the bi-level curriculum of When Priors Backfire — and without the secrecy assumption that Robustness Tokens needs, since backbones are public by construction.

The Defense: Feature-Collapse Noise

Let $\Phi_k$ denote frozen public backbone $k$ (e.g., DINOv2, OpenCLIP, DeiT), $\mu_k$ the mean embedding of the unprotected dataset under $\Phi_k$, $T$ a distribution of trainer-side transformations (crops, color jitter, JPEG compression), and $\epsilon$ the imperceptibility budget. For each image $x_i$ we solve

$$\min_{\|\delta_i\|_\infty \le \epsilon}\; \sum_k w_k\, \mathbb{E}_{t \sim T}\big[\,\|\Phi_k\big(t(x_i + \delta_i)\big) - \mu_k\|^2\,\big],$$

a single-level constrained problem: only $\delta_i$ is optimized, every network weight is frozen. The expectation over $T$ is a cheap Expectation-over-Transformations term, borrowed in spirit from Robust Unlearnable Examples, so the noise survives augmentations and compression rather than living only in high frequencies. Three design points matter. First, the anchor is class-agnostic: unlike error-minimizing noise (Unlearnable Examples: Making Personal Data Unexploitable) or the incorrect-target bindings of When Priors Backfire, the perturbation carries no label information, so it does not create the linearly separable shortcut that a two-layer canary detects with near-100% accuracy (Detection and Defense of Unlearnable Examples). Second, a label-free feature collapse attacks representation reuse, not just classification, addressing the acknowledged gap that classification-bound protections leave segmentation and self-supervised exploitation open. Third, for scrapers who unfreeze the backbone, we add a parameter-noise ensemble — averaging the loss over small random perturbations of $\Phi_k$'s weights — reusing the parameter-neighborhood idea of Provably Unlearnable Examples at forward-pass cost rather than surrogate-training cost.

Research Program

Phase 1 is the measurement the field is missing (Gap 1 in the literature): a grid of scraper adaptation strategies — frozen backbone with linear head, prompt tokens, small adapters (the Robustness Tokens recipe of ~10 tokens trained in ~200 steps), full fine-tuning from pretrained weights, and training from scratch — run against every major protection: EM noise, REM, DUNE, BAIT, and PUE, on CIFAR-10, CIFAR-100, and an ImageNet-100 subset. This tells us how much of the protection survives at each attacker price point, and it motivates the defense by testing whether the cheapest strategies are already sufficient. Phase 2 builds and tunes the feature-collapse noise: backbone-ensemble weighting, patch-level collapse (for segmentation exploiters), transformation set $T$, and the parameter-noise radius. Phase 3 is an adaptive evaluation following the lesson of Optimizing Adaptive Attacks against Content Watermarks for Language Models — static testbeds certify nothing — so the scraper is given full white-box knowledge of the noise recipe and the entire toolbox of counter-measures from APBench: A Unified Benchmark for Availability Poisoning Attacks and Defenses: JPEG and bit-depth reduction, diffusion purification (AVATAR), UEraser, SDA+AN, and Progressive Staged Training from Unlearnable Examples Give a False Sense of Data Privacy.

Baselines

The do-nothing reference points bracket the problem. On clean CIFAR-10, a normally trained ResNet-18 reaches 93.20% (Unlearnable Examples Give a False Sense of Data Privacy), while the same training on EM-poisoned data yields 21.24% (Learning the Unlearnable: Adversarial Augmentations Suppress Unlearnable Example Attacks); no reported result found on Lacuna for a frozen-backbone linear probe on clean or poisoned CIFAR-10, so Phase 1 must establish both numbers itself. The strongest existing competitor for our exact threat model is BAIT (When Priors Backfire), which is the only prior protection aimed at pretrained exploiters and holds a fine-tuned ResNet-18 to 14.40% on CIFAR-10 and Tiny-ViT to 23.64% on CIFAR-100. We will re-run BAIT from its published recipe on identical data and attack grids, with the same perturbation budget and the same tuning effort, and count our method as competitive only if it holds adapter-based scrapers at or below BAIT's levels while cutting generation compute by an order of magnitude (its bi-level meta-optimization plus three-stage curriculum is the cost we target) and additionally resisting strategies BAIT does not address (representation reuse, prompt tuning, detection canaries). A second competitor is REM (Robust Unlearnable Examples), the strongest protection against full adversarial training, which caps accuracy at 48.16% under training-time PGD at radius 4/255 but costs over 50 GPU hours on an ImageNet-100 subset versus 3.9 hours for plain EM noise; we run REM at its published settings and compare cost-accuracy frontiers directly. On the attacker side, the strongest cheap counter-measures are UEraser (restores EM-poisoned accuracy to 95.24%, Learning the Unlearnable) and Progressive Staged Training (restores REM-poisoned accuracy from 21.86% to 78.39%, False Sense of Data Privacy); our noise must degrade these numbers substantially or the defense is hollow. Finally, one ablation isolates the key ingredient: the same $\epsilon$ and the same transformation ensemble, but with the collapse target replaced by a from-scratch surrogate classifier's loss (EM-style) — if the frozen-backbone targeting is what matters, this ablation should fail against adapter scrapers while our method holds.

Why not just…?

Why not just let scrapers use parameter-efficient fine-tuning and declare unlearnable examples dead? For standard EM-family noise this is likely true — pretraining already recovers 61.82–81.40% (When Priors Backfire) and Phase 1 will quantify the adapter version. But "dead" is the wrong conclusion: it means the defense was aimed at the wrong target. Our contribution is precisely the re-aimed defense, so this alternative motivates rather than replaces the work; if our defense also fails, the honest outcome is a well-documented negative result plus the first systematic cost-bypass measurement, which is still worth publishing.

Why not just purify the data (JPEG, or AVATAR diffusion purification)? Purification is the scraper's cheapest realistic response, and APBench shows defenses restore 80–92% accuracy on poisoned data (APBench). Two things cut against it being sufficient: AVATAR multiplies preprocessing cost by the reverse-diffusion steps per image, which contradicts the low-overhead scraping this proposal is about, and low-frequency perturbations such as DUNE's survive filtering and hold accuracy to 14.95–50.82% across seven defenses (Dual-branch Robust Unlearnable Examples). Our noise is not inherently high-frequency (the collapse objective does not prescribe a spectrum), and the $T$ ensemble includes JPEG; still, we treat AVATAR as a decisive check rather than assume victory.

Why not just mix in 20% clean data and retrain? APBench documents that partial clean data collapses existing protections (APBench), and this is the most serious threat to any feature-space method, because clean images would keep their informative embeddings while poisoned ones collapse, giving the head a separable subset. We will measure accuracy as a function of clean-fraction from 0 to 20%; if the defense fails at 5% clean data, we will say so plainly, since within the stated threat model (all published data protected) the attack requires clean data the scraper may not have.

Why not just switch to a different backbone than the ones the defender poisoned? This is the sharpest adaptive response and we do not argue around it: backbones are interchangeable and the defender cannot cover all of them. The mitigation is the ensemble over a backbone zoo plus the known transferability of representation-level attacks — Neural Tangent Generalization Attacks showed architecture-agnostic generalization failure is achievable, and Self-Ensemble Protection showed cheap ensembling improves transfer. Our decisive experiment trains the collapse noise on $N-1$ backbones and evaluates held-out ones; if held-out transfer is poor, the defense's scope shrinks to protecting against the model families that dominate actual scraping, which we will report as a limitation rather than hide.

Proposed Experiments

Datasets and protocols follow the file's benchmarks: CIFAR-10, CIFAR-100, and an ImageNet-100 subset, using APBench's defense pipelines as the evaluation harness so results are directly comparable to its reported restoration range of 80–92% (APBench). Primary metrics: scraper test accuracy under each adaptation strategy (target: at or below BAIT's 14.40% on CIFAR-10 and 23.64% on CIFAR-100, i.e., near the 10% and 1% chance floors); defender generation cost in GPU hours (target: under one GPU hour for ImageNet-100, versus over 50 for REM and roughly 45 for PUE); imperceptibility held at a fixed $\ell_\infty$ budget shared with all baselines; and canary detectability, where we aim to stay far below the near-100% detection rate that linear shortcuts suffer (Detection and Defense of Unlearnable Examples). Decisive checks, in order of importance: held-out-backbone transfer; AVATAR and JPEG purification within the attack loop; clean-fraction sweeps to 20%; Progressive Staged Training and UEraser applied to our noise; and a patch-level variant evaluated on representation reuse. The trade-off lens runs throughout: we track whether the scraper's counter-measures, applied blindly to benign data, degrade clean accuracy — the pattern UEraser documents when aggressive augmentation costs convergence speed, and the general tension formalized in Theoretically Principled Trade-off between Robustness and Accuracy.