Routing to the Clean Expert Only When It Is Safe

A standard classifier trained to resist small adversarial perturbations (worst-case input changes too small for a human to notice) usually loses several points of accuracy on ordinary, unperturbed images. One explanation is that a single set of weights is pulled in two incompatible directions: the gradient that improves clean accuracy and the gradient that improves robustness point increasingly apart as the perturbation budget grows. Conditional computation seems to offer a way out. A mixture-of-experts model contains several sub-networks (experts) and a small gating network, the router, that picks which expert processes each input. With that structure, clean-accuracy weights and robustness weights no longer have to be the same parameters.

The difficulty is that any router choosing between a "clean" expert and a "robust" expert is effectively an adversarial-example detector, and an attacker can aim at the detector. This proposal builds a two-expert model on a shared trunk. Its router is trained asymmetrically: it may send an input to the accurate but fragile clean expert only where that expert's prediction stays correct under every allowed perturbation, and everything else goes to the robust expert. The claim can be tested directly. If the router can be made robust in this one-sided sense, the whole system keeps the robust expert's worst-case accuracy and recovers much of the clean expert's accuracy, at the inference cost of a single expert. If it cannot, adaptive attacks will show it: robust accuracy will fall toward the clean expert's near-zero value on exactly the inputs the router trusted.

Core Claim

A router trained with a one-sided adversarial objective can raise clean accuracy above an adversarially trained model while keeping its robust accuracy, under attacks that target the router and the experts jointly. The router's objective is to never trust the clean expert on a point where the clean expert can be broken. It is not trying to label inputs as clean or perturbed. This is a weaker requirement than general adversarial detection, and that is why it might be achievable. On clean inputs far from the decision boundary, the clean expert is often correct throughout the whole perturbation ball anyway. Those inputs are the ones the router needs to identify.

Relation to Prior Work

Several papers motivate the idea by showing that mixture-of-experts structure interacts with robustness.

  • On the Adversarial Robustness of Mixture of Experts shows that vision mixtures reach better robustness per FLOP than dense ViTs. It also finds that attacks flip routing decisions in deep layers while predictions often survive, because experts overlap in function.
  • Robust Experts finds that adversarially trained CNN mixtures beat dense ResNets on both axes on CIFAR-100. It also finds that a collapsed single-expert path can be up to 7.5 points more robust than full gating, and it names "robustness-aware gating" as open.

Both papers use content-based routers that know nothing about robustness. We make the router's objective be about robustness.

Three papers already split clean and robust computation, and they are the closest competitors.

  • SMART (Sparse Mixture Once-for-all Adversarial Training) uses a shared early trunk with two fixed sparse late paths and separate normalization. A user-set slider chooses the trade-off. We reuse its shared-trunk, late-split layout. We replace the slider with a learned, per-input router whose failure mode we attack directly.
  • JTDMoE (Optimizing Robustness and Accuracy in Mixture of Experts: A Dual-Model Approach) blends the outputs of two full mixtures with a fixed weight. This doubles parameters and inference cost, and the same weight applies to every input. Its diagnosis that experts, not routers, are the weak point under standard attacks (3.11% vs about 55% on CIFAR-10) supports our design. It suggests a router can be hardened more cheaply than an expert.
  • Synergy-of-Experts (Synergy-of-Experts) selects the most confident sub-model using a learned evaluator head. It runs every sub-model at inference and ignores clean accuracy, but it is the closest precedent for learned per-input selection. Its evaluator is exactly the kind of component an adaptive attacker would target, which is why our router is trained against that attacker from the start.

Two papers supply the diagnosis that separate parameters are meant to fix. Conflict-Aware Adversarial Training shows that clean and adversarial gradients diverge as the budget grows. Rethinking Invariance Regularization (AR-AT) attributes part of the clean-accuracy loss to that conflict, and part to batch normalization shared across two input distributions. Helper-based Adversarial Training (HAT) ties the loss to margins that are pushed out further than needed.

Two constraints shape the design:

  • Backdoor routing attacks. Who Speaks for the Trigger? shows that routing logic can be hijacked to reach 100% attack success. This rules out trusting a router that was not trained adversarially.
  • Compute-inflation attacks. ILFO shows that attackers can push adaptive networks toward maximum compute. We therefore give both experts identical FLOPs, so the routing decision cannot be used to inflate cost.

Research Program

The model has four parts:

  • a shared dense trunk covering the early stages;
  • a clean expert $C$ trained with ordinary cross-entropy and its own normalization layers;
  • a robust expert $R$ of equal size, adversarially trained with TRADES-style or HAT-style losses and its own normalization;
  • a router $g(x)\in[0,1]$ attached to trunk features, giving the probability of sending $x$ to $C$. At test time the decision is hard.

The split sits at the last stage. This follows the layer-placement findings of Robust Experts and SMART, which agree that early layers should stay shared. The trunk receives gradients only from $R$'s adversarial loss plus a small clean term. This keeps the early features robust and is where most of the gradient conflict is removed.

The router is trained with an asymmetric objective:

$$\mathcal{L}_g = \mathbb{E}_{x}\Big[\lambda \max_{\|\delta\|\le\epsilon} g(x+\delta)\,\ell\big(C(x+\delta),y\big) \;-\; g(x)\,\big(\ell(R(x),y)-\ell(C(x),y)\big)\Big]$$

The symbols are as follows. $\ell$ is cross-entropy. $\epsilon$ is the threat-model radius. $\lambda\gg 1$ sets how much more a trusted adversarial failure costs than a missed clean gain. The inner maximum is computed jointly over the router and the clean expert, so the training attacker specifically seeks points the router trusts and $C$ gets wrong. The second term pays the router for sending an input to $C$ only when $C$ actually beats $R$ there. The deployed system is robust at $x$ exactly when every perturbed point is either sent to $R$ and classified correctly, or sent to $C$ and classified correctly. For that reason the reported robust accuracy of the system can never be credited above what this per-point condition allows.

Three analyses come with the method:

  1. Where the router sends inputs. We measure, on clean data and under attack, which inputs are routed to $C$ and how that correlates with $C$'s margin. The question is whether the router has learned "the clean expert is locally stable here" or something the attacker can exploit.
  2. Gradient cosine. We measure the cosine between clean and adversarial gradients on shared parameters, with and without the split, following the measurement in Conflict-Aware AT. This tests whether the split removes conflict in the trunk or only moves it.
  3. Trust fraction versus $\lambda$. We sweep $\lambda$ and plot the fraction of inputs routed to $C$. This yields a trade-off curve comparable to SMART's slider, except that each point on it is a per-input policy rather than a global blend.

We extend the design from ResNet to a small ViT, routing per image rather than per token. Per-token routing multiplies the router's attack surface, and choosing per-image routing is itself a stated design decision.

Baselines

Naive reference points. The first is a standard-trained network, which sets the ceiling for clean accuracy. The second is an adversarially trained network of the same inference FLOPs, which sets the floor the method must not drop below. These are also exactly the "always $C$" and "always $R$" settings of our own model, so they cost nothing extra to produce.

Strongest dense trade-off method: AR-AT. On CIFAR-10 with WideResNet-34-10, AR-AT reports 90.89% clean accuracy and 50.77% AutoAttack robustness (Rethinking Invariance Regularization). We also run HAT from its MIT-licensed code (imrahulr/hat). We retrain both on the same backbone, data, and epoch budget as our model, with the same hyperparameter-search budget. Beating AR-AT means higher clean accuracy at equal or better AutoAttack robustness, at matched inference FLOPs, on at least two of CIFAR-10, CIFAR-100, and Tiny-ImageNet.

Strongest routing-style methods: JTDMoE and SMART. JTDMoE reports 92.29% standard accuracy and 74.62% robust accuracy on CIFAR-10 at $\alpha=0.7$ (Optimizing Robustness and Accuracy in Mixture of Experts). SMART reports 95.26% clean and 57.84% robust accuracy with ResNet-34 (Sparse Mixture Once-for-all Adversarial Training). The file does not give the attack protocol behind either robust number. The JTDMoE figure is far above typical AutoAttack results for this budget. We therefore re-evaluate both methods under our full adaptive protocol rather than comparing against the quoted figures, and we charge JTDMoE its doubled inference cost. Beating them means a better clean-versus-AutoAttack curve at no more than their inference FLOPs.

For CIFAR-100 and Tiny-ImageNet, the file contains no isolated numbers for these methods. We report our own reruns.

Key ablations. Four ablations isolate the proposal's ingredients:

  • Symmetric router. The same architecture with a symmetric router ($\lambda=1$, router trained only on clean data) shows whether the one-sided adversarial objective is the ingredient that matters.
  • Shared weights, split normalization. The same two paths with weights tied and only normalization split tests whether the gain is just AR-AT's normalization effect.
  • Detector-gated system. A post-hoc detector trained separately after freezing $C$ and $R$ tests whether training the router jointly with the experts matters.
  • Oracle router. A router that sees the ground-truth perturbation flag gives an upper bound on what any routing policy could achieve.

Why not just…?

Why not just use AR-AT or HAT on a dense network? These methods act on one parameter vector. Conflict-Aware AT reports that clean and adversarial gradients separate as the budget grows, which suggests any single-weight method eventually hits a ceiling. That is a hypothesis, not a settled fact, and it is the main threat to this proposal. The matched-FLOP comparison above decides it. If AR-AT on the same backbone matches our curve, the routing adds nothing and we will say so.

Why not just train an off-the-shelf detector and switch between a clean and a robust model? Separately trained detectors are the standard way such systems fail under adaptive attack. The backdoor-routing results in Who Speaks for the Trigger? show that selection logic becomes the target once it exists. The detector-gated ablation evaluates this system under the same joint attack. Our prediction is that it loses most of its robust accuracy on the inputs it gates to $C$, while the jointly trained asymmetric router does not.

Why not just blend two models, as JTDMoE does, or use SMART's slider? A fixed blend applies the same compromise to every input and, for JTDMoE, doubles inference cost. The honest risk is that a per-input router offers little over a well-chosen global blend. It would only help if the inputs where $C$ is safe are identifiable, rather than scattered at random. The routing analysis measures this directly, by checking whether routing to $C$ tracks $C$'s local margin. If trusted inputs are unpredictable, the router degenerates to a global slider and the proposal fails.

Why not just adversarially train a standard mixture of experts? Robust Experts already does this and gains on both axes. Its routers do not target the trade-off, and its best robust result comes from routing collapse onto one expert. We include its BlockMoE top-2 configuration, adversarially trained, as an additional baseline.

Decisive Checks

We run everything on CIFAR-10, CIFAR-100, and Tiny-ImageNet under $\ell_\infty$ threat models, using ResNet-18, WideResNet-34-10, and a small ViT.

Attack protocol. The primary robust metric is AutoAttack run on the full routed system. During the attack the hard router is relaxed with a straight-through estimator. This is supplemented by three targeted attacks:

  1. one that jointly maximizes $g$ and $C$'s loss;
  2. one that maximizes $R$'s loss subject to staying routed to $R$;
  3. transfer attacks crafted on $C$ alone.

We run the attack-failure indicators from pralab/IndicatorsOfAttackFailure on every evaluation to catch gradient masking caused by the router's discontinuity.

Also reported. For each model we report the fraction of inputs routed to $C$, clean and under attack; inference FLOPs; and robustness at budgets of $1.5\epsilon$ and $2\epsilon$, larger than the training budget.

Outcome criteria.

  • Success: system AutoAttack accuracy within one point of the robust expert alone, and clean accuracy at least three points above it and above AR-AT, at matched FLOPs.
  • Failure: under the joint attack, the router can be pushed to trust $C$ on more than a few percent of attacked inputs. That would mean one-sided robust routing is not easier than detection, which is itself a useful negative result for this line of work.