One Trigger, Every Architecture: Meta-Learned Clean-Label Backdoors Across the CNN–ViT Divide

A backdoor attack plants a dormant trigger in a model by tampering with its training data: the model behaves normally on clean inputs, but any input carrying the trigger is pushed to an attacker-chosen label. The hardest and most realistic version is the clean-label setting, where poisoned images keep their true labels, so neither human inspection nor label-cleaning tools flag them. Two trends now make these attacks fragile in practice, and both are documented in the literature. First, triggers crafted against one model family fail on another: methods built for convolutional networks do not survive contact with Vision Transformers (ViTs) — models that process images as patches with attention rather than with local filters — and vice versa. Second, victims increasingly fine-tune from pretrained checkpoints rather than train from scratch, and pretrained priors wash out synthetic shortcut features, a failure mode demonstrated directly for unlearnable-example attacks by When Priors Backfire.

I propose to treat backdoor trigger crafting as a meta-learning problem over a distribution of victims, rather than an optimization against one surrogate. A trigger generator is trained so that the backdoor it plants survives the training dynamics of many different victim models — convolutional and transformer, from-scratch and pretrained-initialized — and is evaluated during crafting only on held-out architectures it never saw, forcing it to exploit victim-invariant weaknesses instead of one surrogate's quirks. The payoff would be a single, reusable, clean-label trigger that crosses the CNN–ViT divide and survives pretrained fine-tuning, plus a sharp empirical answer to whether meta-learning over victims is genuinely necessary or whether simple surrogate ensembling suffices.

Core Claim

The claim is that the transferability machinery developed for test-time adversarial examples and single-target poisoning — meta-train/meta-test splits over a model zoo — can be moved inside a bi-level backdoor pipeline to produce input-agnostic triggers that no current method can produce. Concretely, a trigger generator $g_\phi$ (a small U-Net, following COMBAT) is optimized so that when a poisoned dataset is built by applying $g_\phi$ to clean target-class images, any victim trained on that dataset acquires the backdoor:

$$\max_{\phi}\; \sum_{j \in \mathcal{V}_{\text{held}}} \text{ASR}\big(f_{\theta_j(\phi)}\big) \;-\; \lambda\, L_{\text{stealth}}(g_\phi),$$

where $\theta_j(\phi)$ denotes the parameters of victim model $j$ after training on the clean data plus the poisoned set generated by $g_\phi$, ASR is the fraction of triggered test inputs misclassified to the target label, and $\mathcal{V}_{\text{held}}$ is a set of architectures disjoint from the surrogate zoo used to compute cheap inner-loop gradients. The stealth term bundles clean labels, a low-frequency Discrete Cosine Transform constraint on the trigger (the frequency-domain smoothing that COMBAT showed defeats frequency-based screening), and a new consistency term asking that triggered images stay close to natural target-class images in an independent clean feature extractor, targeting the detection assumption of NTD. This is an explicitly adversarial-security research contribution: the aim is to measure how far the attack surface extends, in the standard spirit of adversarial robustness evaluation.

The gap this fills is precise. MetaPoison and Witches' Brew achieve cross-architecture transfer, but only for targeted poisoning of one pre-selected test image — they plant no reusable trigger, so the attack dies after a single query. COMBAT and Narcissus plant reusable clean-label triggers, but tie optimization to a single local surrogate and were evaluated only on convolutional networks; COMBAT's authors state ViT extension as open work. Megatron builds ViT backdoors with attention-aligned losses, but only within the ViT family, against a static surrogate feature space, without modeling the victim's training dynamics. And BAIT shows how to fight pretrained priors in a bi-level loop, but for availability protection (destroying clean accuracy), not for backdoors that must preserve clean accuracy. No existing work combines all three: reusable trigger, cross-family meta-transfer, and survival under pretrained initialization.

Research Program

The build has three parts. First, the inner loop trains a zoo of surrogate victims — ResNet-18, VGG, MobileNet-v2 (the architectures used in Witches' Brew) plus ViT-base, DeiT, ViT-small and CaiT (the architectures used in Megatron) — on clean data plus the current poisoned set, with some surrogates initialized from ImageNet-pretrained weights and others from scratch. Second, the outer loop updates the trigger generator: it periodically retrains a held-out proxy subset (architectures excluded from the inner zoo, and held-out initialization regimes) for a few steps on the poisoned set, and updates $\phi$ to raise ASR on those proxies, in the spirit of the disjoint-split meta-gradient attack of MGAA but relocated from test-time perturbations to training-time triggers. To keep this tractable I will use first-order approximations — a few unrolled steps at the start of proxy training, and COMBAT-style alternation elsewhere — rather than the full unrolled SGD that costs MetaPoison roughly 6 GPU-hours for 500 poisons; the hypothesis under test is that the split, not the depth of unrolling, is what buys transfer. Third, the pretrained-prior binding: following the lesson of BAIT — that static shortcut features collapse when victims fine-tune pretrained representations, where baseline unlearnable examples let accuracy recover to 61.82% on CIFAR-10 — the inner zoo deliberately includes pretrained-initialized victims, and a curriculum increasingly weights them, so the generator is pushed toward trigger features that pretrained priors do not overwrite.

Stealth is treated as a constraint set rather than a research contribution of its own: clean labels throughout, the DCT low-frequency bound and Gaussian smoothing of COMBAT, and the clean-extractor consistency term aimed at NTD, which flags inputs whose representations diverge between the victim and an independent clean feature extractor. I deliberately avoid generative latent-space stealth (the diffusion or StyleGAN machinery of CBV and CLIP2Protect): those are heavier, and the lighter DCT route is already proven adequate against frequency screening, reducing from 100% to 16% detection in COMBAT's own evaluation.

Evaluation runs on CIFAR-10, CIFAR-100, Tiny-ImageNet and GTSRB — the four datasets shared by the COMBAT, Megatron and STRIP evaluations — with poison budgets of 0.05%, 1% and 5%. Victims are trained both from scratch and from ImageNet-pretrained initialization, in all four transfer directions (CNN surrogate to ViT victim, ViT to CNN, and within-family). Reported metrics: ASR on triggered test inputs, clean-accuracy degradation, a full CNN↔ViT transfer matrix, PSNR/SSIM/LPIPS image quality, detection rates under STRIP, Fine-Pruning, Neural Cleanse-style inspection and NTD, and total crafting GPU-hours.

Baselines

The do-nothing reference is clean training on unmodified data with the trigger applied only at test time; its ASR should sit near zero, and it anchors the claim that the backdoor comes from the poisoned data rather than from the trigger pattern being inherently confusable. Any leakage above a few percent would mean the trigger is just a natural class correlate and would invalidate the attack framing.

The strongest existing competitor is COMBAT, the alternated generator/surrogate clean-label backdoor that reports 97.73% ASR on CIFAR-10 at a noise budget of $\eta = 10/255$ and 99.84% ASR on CelebA. It is the strongest because it is the only method that already combines a bi-level alternated structure with clean-label stealth and defense evasion; the proposal is essentially COMBAT plus a meta-split, so COMBAT defines the bar. It will be run with its published architecture and losses on the same datasets, poison budgets and victim-training protocols, with an equal GPU-hour tuning budget per method. An important caveat I will resolve empirically rather than assume: no reported result found on Lacuna for COMBAT on ViT victims, so a pilot experiment must first establish how far COMBAT's ASR falls on DeiT or ViT-small before the cross-family comparison means anything. A second strong competitor is Megatron, which reports 96.36% ASR on Tiny-ImageNet in one-to-one attacks and over 90% on CIFAR-100 and GTSRB; it will be run in the reverse direction (ViT-surrogate triggers against CNN victims), where I again expect collapse, with no reported cross-family number existing on Lacuna. Narcissus, at 85.81% ASR on Tiny-ImageNet with a 0.05% poison ratio, serves as the low-information reference point, and its need for test-time trigger magnification is itself a dependent variable I will measure. Beating the baselines means: at matched budget and compute, maintaining at least 90% ASR on held-out cross-family victims while the best baseline drops by at least 30 points, with clean-accuracy loss under two points, and holding that ASR under pretrained-initialization fine-tuning where baselines were never designed to operate.

The key-ingredient ablation replaces the held-out meta-split with plain surrogate ensembling: identical generator, identical losses, identical zoo, but the outer loop simply averages ASR across all inner-loop surrogates with no held-out proxies and no pretrained-weighting curriculum. A second ablation swaps the alternated outer loop for a first-order gradient-matching update in the style of Witches' Brew, aligning poison gradients with the gradients of triggered target-label loss, to test whether the expensive alternation matters once the meta-split is in place. A third ablation drops the clean-extractor consistency term to isolate whether NTD evasion comes from that term or from the low-frequency constraint alone.

Why not just…?

Why not just ensemble CNN and ViT surrogates in the outer loop? This is the most serious alternative, and I treat it as a live hypothesis rather than a strawman. The evidence against it comes from the test-time setting: MGAA showed that averaging losses over surrogates overfits their shared idiosyncrasies, and that a disjoint meta-split raised targeted black-box transfer from 23.86% to 38.18% when integrated with TI-DIM. But no one has shown this carries over to backdoor triggers, where the outer objective is far richer than a single gradient direction; it is entirely possible that ASR across a heterogeneous ensemble is already a strong enough transfer signal. The ensembling ablation is designed to settle exactly this, and if plain ensembling matches the meta-split on the transfer matrix, I will report the meta-split as unnecessary and recenter the paper on the pretrained-survival result — which no ensembling baseline addresses either way.

Why not just use first-order gradient matching instead of alternated training? Witches' Brew shows how dramatic the savings can be: it cut ImageNet-scale poison crafting from an estimated 500+ GPU-days to under 4 hours and beat MetaPoison 90% to 42.5% on ResNet-18. But its objective aligns poison gradients with the gradient of one specific target image — there is no mechanism for planting input-agnostic trigger logic, which is why backdoor work turned to generator-surrogate co-optimization in the first place. Still, a Witches'-Brew-style matching toward a triggered-target loss across many images is a plausible cheap variant, so I am running it as an ablation rather than dismissing it; if it matches the alternated loop, it becomes the default implementation, since the contribution I am claiming is the victim distribution, not the inner-loop machinery.

Why not just keep two triggers, COMBAT for CNNs and Megatron for ViTs? If the attacker knows the victim family, this works today and the practical gain of a single cross-family trigger is small — I say so plainly. The scientific claim is different: a trigger that both a convolutional network and a Vision Transformer acquire from the same poisoned data, while clean accuracy holds, is evidence of a victim-invariant failure mode that per-family tricks do not expose, and it covers the realistic setting where the attacker publishing or leaking a dataset cannot know which architecture will eventually be trained on it. The experiments report per-family and cross-family numbers side by side so readers can judge the practical margin themselves.

Why not just add DCT smoothing and call it stealthy? Essentially, yes — that is what I am doing. The stealth toolkit is borrowed wholesale from COMBAT, and I make no claim of novelty there. The one new stealth element is the clean-extractor consistency term against NTD, whose near-0% online false-acceptance rate on FaceScrub rests on the assumption that triggers do not transfer to clean feature extractors; the ablation dropping this term will show whether it is load-bearing or whether low-frequency triggers already slip past NTD.

Decisive Checks

Four checks decide the proposal. First, the transfer matrix: craft with a CNN-heavy zoo, attack ViT victims, and vice versa; the pilot must first show the baseline collapse (COMBAT and Megatron dropping sharply outside their families), since no cross-family numbers exist on Lacuna for either. Second, pretrained survival: victims fine-tuned from ImageNet-pretrained weights, where BAIT proved static shortcuts fail — the analog here is watching whether baseline-trigger ASR collapses under pretrained fine-tuning while the meta-learned trigger holds, mirroring how BAIT held CIFAR-100 accuracy to 23.64% where baseline unlearnable examples allowed 81.40%. Third, defense stack: STRIP, which reports 0% false-acceptance and 0% false-rejection on CIFAR-10 and GTSRB; Fine-Pruning, which in its own setting cut a face-recognition backdoor from 100% ASR to 0%; and NTD with its clean-extractor comparison. Fourth, cost honesty: report crafting GPU-hours against MetaPoison's 6 GPU-hours per 500 poisons and MGAA's 67-seconds-per-image overhead, because a method that only transfers at unaffordable cost is a negative result worth knowing. If the meta-split beats ensembling on the transfer matrix and holds ASR under pretrained fine-tuning, the core claim stands; if ensembling ties, the paper becomes the first controlled study showing that backdoor transfer, unlike test-time transfer, does not need meta-learning — either outcome is publishable, and the experiments are built to distinguish them cleanly.