Closing the Doors the Attacker Uses: Robust-Context Channel Gates for Adversarial Training

Adversarially trained image classifiers are trained against worst-case small input perturbations. They pay for their robustness with clean accuracy. They throw away the fine texture cues that ordinary networks rely on, because those cues are exactly what a small perturbation can corrupt. This proposal tests a specific way to recover some of those cues. A small gate decides, for each input, which internal feature channels are allowed to pass. The gate reads only a coarse, deliberately hardened summary of the image, and it is trained so that an attacker cannot change its decision. The intuition comes from work on context-aware feature filtering in language models, where a sentence-level summary multiplies each word's embedding by a vector of values between 0 and 1, silencing the dimensions that belong to the wrong sense of an ambiguous word (Handling Homographs in Neural Machine Translation, Selective Encoding for Abstractive Sentence Summarization). Here, the "wrong sense" is the set of channels that would argue for the wrong class.

The opportunity rests on one empirical finding and one gap. The finding comes from CIFS: attacks succeed largely by inflating "negatively relevant" channels, meaning channels whose presence argues against the true class. The gap is that every existing filter computes its decision from features the attacker can already move. CIFS uses gradients of an intermediate probe classifier. Feature Denoising uses self-attention over the same perturbed feature map. If the gate itself can be attacked, filtering only relocates the vulnerability. The proposal is to make the gate's input and decision provably hard to move, and then to measure whether a gate that cannot be reopened lets the main network keep its fragile but accurate channels.

Core Claim

A channel gate conditioned on a robust coarse context, and trained to keep its decision stable within the perturbation budget, will improve clean accuracy at matched AutoAttack robustness relative to TRADES and HAT on CIFAR-10 and CIFAR-100. A gate of identical architecture whose context comes from full-resolution, unhardened features (the obvious "gating plus adversarial training" design) will not. The new ingredient is not gating itself. It is the separation between a hard-to-move signal that decides which channels open and an expressive signal that flows through them, plus a direct measurement of whether attackers flip the gate.

Concretely, let $x_\ell$ be the feature map at layer $\ell$ of the main network. Let $c = E(\mathrm{down}(x))$ be a context vector produced by a small encoder $E$ from a heavily downsampled copy of the input. The gated features are

$$\tilde{x}_\ell = x_\ell \odot \sigma(W_\ell c + b_\ell),$$

where $\sigma$ is the logistic function and $W_\ell, b_\ell$ are per-layer gate parameters. Here $\odot$ multiplies each channel of $x_\ell$ by its gate value. The full objective is

$$\mathcal{L} = \mathcal{L}_{\mathrm{TRADES}}(f) + \gamma \, \mathbb{E}_x \Big[ \max_{\|\delta\|_\infty \le \epsilon} \sum_\ell \big\| g_\ell(x+\delta) - g_\ell(x) \big\|_1 \Big] + \mathcal{L}_{\mathrm{TRADES}}^{\epsilon'}(E),$$

where $f$ is the whole classifier and $g_\ell$ is the gate vector at layer $\ell$. The first term is the TRADES loss of Theoretically Principled Trade-off between Robustness and Accuracy. The second term is a gate-stability penalty with weight $\gamma$, found with its own inner attack. The third term adversarially trains the context encoder, through a small auxiliary head, at a larger budget $\epsilon' > \epsilon$.

Downsampling alone does not buy robustness, since a constant-sign $\ell_\infty$ perturbation survives average pooling intact. That is why the encoder is hardened explicitly rather than assumed robust.

Why This Reconciles a Real Tension

Robust Local Features for Improving the Generalization of Adversarial Training argues that adversarial training over-relies on global shape, which is robust but generalizes poorly. Its remedy is to destroy global context by shuffling image patches. Feature Denoising finds the opposite: aggregating global context beats local filters for suppressing adversarial noise. The design here takes a position between them. Global structure is used only to decide which local channels are admissible, never as the evidence for the class itself. Evidence still comes from local, texture-sensitive channels, which addresses Song et al.'s generalization concern. The robust global signal is what keeps the attacker from opening wrong-class channels, which is consistent with Xie et al.

This also gives a mechanistic reading of the excessive-margin finding of HAT. HAT shows that adversarial training pushes boundaries unnecessarily far and discards fine features, and it fixes this in the loss with "helper" examples. We attempt the fix inside the representation. Fine channels are kept, but they are only reachable when the robust context says they are relevant.

Research Program

The first stage is diagnostic and can falsify the premise cheaply. Using standard PGD-adversarially-trained and TRADES ResNet-18 models on CIFAR-10, we reproduce the CIFS-style split of channels into positively and negatively relevant. We then ask how well a linear probe on a hardened 8×8 context can predict, before seeing the full image, which channels will be negatively relevant. If the coarse context predicts relevance poorly, the gate has nothing to work with and the project stops there.

The second stage trains the gated network end-to-end with the objective above. Gates sit at the outputs of three residual stages, so filtering happens early as well as late. This is motivated by Masking Adversarial Damage, which finds that early layers carry the highest adversarial sensitivity. It also contrasts with Less is More, which masks only the penultimate layer and needs a test-time counter-attack.

We also test whether that paper's stability-based mask can serve as a distillation target for the gate. If it can, its feature-selection signal would be obtained in a single forward pass with no test-time optimization.

The third stage is analysis. We measure the gate-flip rate under attack, and check whether CIFAR-100 gates organize around its coarse superclasses without being told about them. We also track whether robust overfitting, which CIFS reports its masks mitigate, is likewise reduced.

Training and evaluation build on the official HAT codebase, imrahulr/hat. It already implements standard adversarial training, TRADES, HAT and AutoAttack evaluation on CIFAR-10, so every baseline shares one data pipeline and one attack implementation.

Baselines

The naive reference point is standard PGD adversarial training on ResNet-18. Less is More reports 47.33% robust accuracy for it on CIFAR-10, and CIFS reports 44.72% under PGD-100. We rerun it ourselves rather than relying on either figure.

TRADES is the loss our method builds on. Its reported white-box robust accuracy reaches 56.61% (TRADES) under a different setup, so the comparison that matters is our own rerun. Our method minus the gate is exactly TRADES, which makes this the cleanest point of comparison.

The strongest existing competitor on the accuracy-robustness question is HAT. Reducing Excessive Margin reports roughly 49% AutoAttack accuracy on PreAct ResNet-18 on CIFAR-10, with clean accuracy about 2.17 points above TRADES. On CIFAR-100 it reports a clean gain of about 2.49 points. HAT targets the same failure we target, discarded fine features, but through the loss. Beating it means higher clean accuracy at AutoAttack robustness within 0.5 points of HAT's, or higher AutoAttack robustness at matched clean accuracy. We also run HAT plus our gate, because the two mechanisms may compose.

The strongest feature-selection competitor is CIFS. It reports 48.70% versus 44.72% under PGD-100 for ResNet-18 on CIFAR-10, at a cost of 1 to 2 points of clean accuracy. We rerun it under AutoAttack, since its mask is computed from attackable probe gradients and its PGD number may overstate robustness.

We do not treat as competitive targets the very high PGD-20 figures reported by M2AT (80.66% on WRN-34-10) or LM-CCA (63.4%). Numbers that far above AutoAttack-evaluated methods usually indicate a weak evaluation. We rerun both under AutoAttack only as a sanity check.

Two ablations isolate the key ingredient. The first gives the same gate a context taken from the main network's own pooled full-resolution features, with no hardening and no stability term. This is essentially squeeze-and-excitation-style channel gating trained with TRADES, and it is the obvious version of this proposal. The second keeps the robust context but removes the gate-stability term. A third control is a width-matched TRADES network with the same parameter count and the same training FLOPs, including the extra inner attack, so that gains cannot be explained by capacity or compute.

All methods share the same data, a 200-epoch schedule (sized for the fixed budget), $\epsilon = 8/255$, an equal hyperparameter search of 12 configurations each, and three seeds. Checkpoints are selected on a held-out validation split using PGD robust accuracy.

Why not just…?

Why not just add squeeze-and-excitation blocks and run TRADES? This is the most serious threat. Squeeze-and-excitation blocks, which rescale channels from a global average of the same feature map, are already a context-to-channel gate. If they deliver the same gains, the idea reduces to a known module, and we would say so. Our hypothesis is that their context is computed from the very features the attacker perturbs, so the attacker can push gates open on wrong-class channels. That is CIFS's channel-hijacking mechanism, now acting through the gate. The first ablation in Baselines is exactly this model. The deciding measurements are AutoAttack accuracy, an adaptive attack that maximizes gate change, and gate-flip rate.

Why not just use HAT, which already fixes excessive margin? HAT recovers clean accuracy through helper examples, without changing what the network computes (Reducing Excessive Margin). It may simply be sufficient. If our gated TRADES model fails to beat HAT at matched robustness and HAT plus gate also fails to improve on HAT, the representational route adds nothing. We would report that outcome as a clean negative result.

Why not just use CIFS, which already masks hijacked channels? CIFS supplies the core evidence for the approach, and it is also the method to beat on mechanism. Its mask comes from first-order gradients of a probe that reads perturbed intermediate features, so in principle an adaptive attacker can steer the mask itself. Our design replaces that with a learned, feed-forward gate whose input is hardened. Whether this matters empirically is settled by running both under the same AutoAttack and gate-targeted attacks. If CIFS's mask turns out to be just as stable, our contribution narrows to cost and simplicity.

Why not just denoise the input before classifying? AdvFilter documents that additive input denoisers over-smooth clean images and break across perturbation strengths. Feature Denoising shows the corruption lives in internal features. We include no separate input-denoising baseline beyond those reports, and we do not claim to have tested this option ourselves.

Proposed Experiments

The main experiments use ResNet-18 and PreAct ResNet-18 on CIFAR-10 and CIFAR-100. We then run a confirmation on WRN-34-10 and on STL-10, where RLFAT reports TRADES improving from 65.24% to 72.38% clean and from 38.99% to 43.36% PGD robust accuracy. That makes STL-10 the direct test of our reconciliation claim against the patch-shuffling remedy.

The metrics are clean accuracy and AutoAttack accuracy at $\epsilon = 8/255$. We also run an adaptive attack whose loss adds a term that maximizes gate change, and we approximate gradients straight through the gates if they saturate. Further measurements are gate-flip rate under each attack, robust-accuracy curves across epochs to detect robust overfitting, and training and inference cost relative to TRADES.

Three checks decide the outcome. First, the robust-context gate must beat the full-resolution-context ablation under AutoAttack, and must show a substantially lower gate-flip rate. Without that, the central claim fails. Second, the adaptive gate attack must not reduce robust accuracy by more than one point below plain AutoAttack. A larger drop would indicate the gate is masking gradients rather than adding robustness. Third, the clean-accuracy gain over HAT at matched AutoAttack robustness must exceed seed variance on both CIFAR-10 and CIFAR-100. On CIFAR-100 we additionally test whether the gate patterns cluster by superclass. That would be direct evidence that the gate is doing context-dependent selection rather than generic rescaling.