Adversarial training makes a classifier resist small, deliberately crafted input changes. It does this by attacking the model during training and learning from the attacks. That makes it expensive: every update first runs an iterative attack, typically ten or more extra forward and backward passes. Mixture-of-experts models make the cost worse. These models send each input to one of several specialized sub-networks ("experts") chosen by a small selector network (the "router"). The best published recipes for making them robust train the router and experts in alternation or in several stages per epoch, and they pay for it. AdvMoE reports roughly twice the wall-clock time of ordinary adversarial training (Robust Mixture-of-Expert Training for Convolutional Neural Networks). The video method J-TLAT runs three attack-bearing steps every epoch (Exposing and Defending the Achilles' Heel of Video Mixture-of-Experts).
We propose to remove the joint optimization that causes this extra cost. Each expert is adversarially trained on its own fixed slice of the data, with no router present. A router is then fitted afterwards over the frozen robust experts. It is trained on attack examples saved during expert training, so fitting it requires no new attack passes. The bet is that "router–expert inadaptability" is a symptom of training the two jointly under attack, which AdvMoE identified as the reason plain adversarial training fails on mixture models. If so, decoupling removes the need for alternation. The result would be a robust mixture at about the cost of one ordinary adversarial training run.
Core Claim
Assume the experts are built on a shared, frozen backbone and the router is fitted only after the experts are finished. Then a mixture model can reach the robust accuracy of alternating schemes like AdvMoE for roughly half their attack-pass budget. The total training cost reduces to one pass of adversarial training over the dataset, split across experts that can run in parallel, plus a router stage that adds no new attacks.
Three findings make this plausible.
First, Optimizing Robustness and Accuracy in Mixture of Experts: A Dual-Model Approach compared attacks on each component alone on CIFAR-10. Attacking only the experts drops accuracy to 3.11%, while attacking only the router leaves about 55%. Robustness effort therefore belongs mostly in the experts, and those are exactly what our first stage trains.
Second, On the Adversarial Robustness of Mixture of Experts found that attacks often flip routing decisions in a trained mixture without changing the final prediction. Experts overlap enough in what they can handle that a wrong routing choice is often harmless.
Third, decoupling already works for clean training. Two families of methods show this. In expert-first methods, experts are trained in isolation on separate data slices and frozen before any router is trained: Self-MoE trains small low-rank adapter (LoRA) experts and then a router over them (Self-MoE), and TimeStep Master trains one expert per noise interval of a diffusion model before assembling them (TimeStep Master). In router-first methods, the routing decisions are fixed early and held constant: StableMoE and Grouter. None of these four papers involves an adversary. We borrow the expert-first recipe from them, and specifically not the Grouter recipe, which depends on a large trained teacher model to supply its routing.
Research Program
Expert unit. Each expert is a parameter-efficient adapter on a frozen pretrained Vision Transformer. We use the LNLoRA module from FullLoRA-AT, which adds a learnable layer normalization in front of each low-rank adapter. That paper showed ordinary LoRA fails to absorb adversarial robustness, and the learnable normalization is what fixes it.
Stage 0: partitioning. Training images are clustered on clean features from the frozen backbone into $K$ groups, following the semantic clustering of CLIP-FMoE. The clustering costs only one clean forward pass over the data.
Stage 1: expert training. Expert $k$ is trained with standard projected-gradient-descent attacks (PGD, an iterative attack that repeatedly steps along the gradient sign within the allowed perturbation budget) using only its own cluster. Because each image belongs to one cluster, total attack cost equals a single run of adversarial training over the dataset. The experts share no gradients, so they train in parallel. Every final PGD perturbation is stored together with its image.
Stage 2: router fitting. All experts are frozen. A small router reading early backbone features is trained on clean images and on the cached perturbed images. It has two objectives. The first is to send each input to an expert that classifies it correctly. The second is a KL-divergence consistency term that keeps routing on perturbed inputs close to routing on the clean version, in the spirit of the second-best-expert loss in RT-ER. Stage 2 runs no new attacks.
Stage 3 (optional): adaptive refresh. A short refresh of the router against fresh attacks aimed at the router. This exists only if the evaluation in the experiments section shows the cached perturbations are stale. Its cost is reported separately.
Cost accounting. All costs are counted in total forward-plus-backward passes and in GPU-hours, including the clustering in Stage 0.
Transfer check on convolutional networks. A second, smaller study repeats the recipe for a ResNet-18. A robustly trained dense checkpoint supplies the shared trunk. Its deepest stage, conv5_x, is copied into $K$ experts, which are then trained per cluster. We chose that stage because Robust Experts found mixture layers there give the most robustness per parameter. This setup allows a direct comparison with AdvMoE's own architecture family. The trunk's training cost is included in our budget.
Baselines
The naive reference is end-to-end PGD adversarial training of the same mixture architecture: router and experts updated together, with the same adapter experts and the same data. This is the recipe used by On the Adversarial Robustness of Mixture of Experts and by Robust Experts. AdvMoE argues it underperforms because of router–expert inadaptability. Our method beats it if it matches or exceeds its AutoAttack robust accuracy at an equal or smaller pass budget.
The strongest existing method is AdvMoE (Robust Mixture-of-Expert Training for Convolutional Neural Networks), which alternates router and expert updates under attack. Its reported cost is about 2× ordinary adversarial training. The research file contains no robust-accuracy figure for AdvMoE: no reported result found on Lacuna for AdvMoE robust accuracy on CIFAR-10 or ImageNet. We will therefore rerun it ourselves.
The rerun will use the same backbone, experts, data splits, attack strength, and hyperparameter search budget, in two settings. In the first, AdvMoE gets its full ~2× budget. In the second, it is truncated to our budget. Our method beats AdvMoE if it is within one point of AdvMoE's full-budget robust accuracy at roughly half the passes, or strictly better at the matched budget.
J-TLAT is included in the convolutional study as a costlier upper reference. The Dual-Model method JTDMoE is included where its two-model storage fits.
The dense competitor is FullLoRA-AT itself: a single robust adapter trained on all the data with the same total number of attack passes. The file reports its cost as about 75% of full adversarial training's time and 5% of its trainable parameters, but gives no robust-accuracy number for it here.
Two ablations isolate what matters:
- Partition ablation. Random partitions in place of semantic clusters. This tests whether specialization matters or whether the gain is only ensembling.
- Router ablation. The router trained with the cached perturbations versus with clean data only. This tests whether reusing saved attacks is what makes the router robust.
Why not just…?
Why not just adversarially train one dense adapter, since collapsing to a single expert was the most robust in prior work? This is the most serious threat. Robust Experts found that when routing collapsed onto a single expert, that sub-network was up to 7.5 points more robust than the full mixture. Part of that gap may come from jointly trained routers being unstable under attack, which our design removes. That is not guaranteed, though. If the dense FullLoRA-AT baseline matches us at equal passes, our contribution reduces to a cost-equivalent mixture with no robustness benefit. We would report it that way, and the claim would narrow to the clean-accuracy and capacity gains of having several experts.
Why not just freeze a router learned on clean data, StableMoE-style, and then adversarially train all experts jointly? This option keeps the stable routing but still trains the experts jointly. It costs the same as our method, because one attack per example is run either way. What differs is whether experts see only their own cluster. We include it as a third arm. If it ties our method, then decoupled routing alone is the active ingredient, and per-cluster isolation is unnecessary.
Why not just make AdvMoE cheaper with single-step attacks? Fast single-step adversarial training would cut AdvMoE's inner attack loop. The same trick applies equally to our Stage 1, so it is a separate dimension of savings rather than a competitor. The comparison that isolates our contribution is therefore the one at an equal attack type. We also run both methods with single-step attacks to check that the savings from decoupling and from cheaper attacks stack. No result for single-step AdvMoE appears in the file.
Why not just train the router under fresh attacks, as AdvMoE and J-TLAT do? J-TLAT's router-specific attack drops robust accuracy to 18.13%, and its joint router-plus-expert attack drops it to 2.54%. These numbers show routers can be a real attack surface, so the saved Stage 1 perturbations may not be enough to protect the router. Stage 3 exists to measure exactly how much fresh router-side attacking is needed, and its cost is charged against our budget.
Proposed Experiments
The main benchmarks are CIFAR-10, CIFAR-100, and Tiny-ImageNet for the Vision Transformer adapter study. An ImageNet run is added if budget allows; these are the datasets AdvMoE uses. The convolutional study uses ResNet-18 on CIFAR-10 and CIFAR-100. All runs use $\ell_\infty$ perturbations with radius $8/255$. Metrics are clean accuracy, PGD-20 accuracy, AutoAttack accuracy, and total passes and GPU-hours.
The headline result is a curve of robust accuracy against training passes for every method. Our claim succeeds if our method lies on or above AdvMoE's curve while ending at about half its cost.
Decisive check: adaptive attacks. Because our router is fitted to attacks against experts, an evaluation using only those attacks would flatter it. We therefore run three component attacks on the final model, following the Dual-Model and video papers:
- an expert-only attack,
- a router-only attack that pushes inputs toward each input's least-confident expert,
- a joint attack on router and experts together.
If the joint attack opens a gap of more than a few points relative to AdvMoE, the cached-perturbation router has failed and Stage 3 becomes mandatory. In that case, the reported savings must include Stage 3's cost.
Second check: specialization. We test whether experts trained in isolation are actually needed by sending each input to a randomly chosen wrong expert. A large drop shows the experts specialized. No drop means the result is ensembling rather than specialization.
References
- Robust Mixture-of-Expert Training for Convolutional Neural Networks
- Exposing and Defending the Achilles' Heel of Video Mixture-of-Experts
- Optimizing Robustness and Accuracy in Mixture of Experts: A Dual-Model Approach
- On the Adversarial Robustness of Mixture of Experts
- Self-MoE: Towards Compositional Large Language Models with Self-Specialized Experts
- TimeStep Master: Asymmetrical Mixture of Timestep LoRA Experts for Versatile and Efficient Diffusion Models in Vision
- StableMoE: Stable Routing Strategy for Mixture of Experts
- Grouter: Decoupling Routing from Representation for Accelerated MoE Training
- FullLoRA-AT: Efficiently Boosting the Robustness of Pretrained Vision Transformers
- CLIP-FMoE: Scalable CLIP via Fused Mixture-of-Experts with Enforced Specialization
- Robust Experts: the Effect of Adversarial Training on CNNs with Sparse Mixture-of-Experts Layers