Designing Accelerators for Adversarial Training, Not Just Inference

Adversarial training teaches a network to resist small malicious input changes. It does so by solving a nested problem: an inner loop searches for the worst perturbation of each input, and an outer loop updates the weights on those perturbed inputs. The inner loop multiplies training cost many times over. Fast single-step variants cut that cost on GPUs, but they are fragile. Meanwhile, automated hardware design space exploration (DSE) now searches millions of accelerator configurations using fast cost models. DSE here means algorithmically choosing processing-element counts, buffer sizes, and loop tilings for a custom chip. Every DSE tool in this literature models only the forward inference pass. We propose to build the missing piece: a DSE cost model and search that treats an adversarial training step as its own workload. The model separates "attack passes," which backpropagate only to the input, from "weight passes," which also compute weight gradients. The search then chooses the hardware and the training recipe together, so the resulting design minimizes the energy needed to reach a target robust accuracy.

The core bet is that the two kinds of pass want different hardware. Attack passes skip the weight-gradient work and the optimizer state. Weight passes need both. The recipe fixes the ratio between them: PGD-$K$ (K attack steps per weight update), single-step Fast AT, or Free AT with $m$ minibatch replays. So the best buffer-to-compute ratio should shift with the recipe, and a chip tuned for inference, or for one recipe, should be measurably off for the others. If that shift turns out to be negligible, the co-design claim is wrong, and the experiments are built to show it.

Core Claim

The design question should be posed over pairs: an accelerator configuration $h$ and an adversarial-training recipe $a$. The recipe covers attack steps $K$, replay factor $m$, whether random initialization is used, and arithmetic precision. We seek

$$\min_{h,a} \; E(h,a) \quad \text{s.t.} \quad R(a) \ge r,\;\; \text{Area}(h) \le A,$$

where $E(h,a)$ is the modeled energy (or wall-clock time) to complete training with recipe $a$ on hardware $h$. $R(a)$ is robust accuracy under AutoAttack, $r$ is a target, and $A$ is an area budget.

The formulation factorizes, and that is what makes it tractable. Robustness depends on the recipe and its numerics, not on the chip. So $R(a)$ can be measured once per recipe on a GPU with emulated precision. The large combinatorial search happens only in $E$, where a fast analytical model makes evaluating millions of points cheap.

What is new relative to existing tools is the following combination:

  • a cost model with explicit pass types and tensor lifetimes for min-max training;
  • robust accuracy, not throughput, as the constraint the hardware search serves;
  • precision treated as a joint variable, because it can trigger catastrophic overfitting (training that appears robust against the single-step attack but collapses to near-zero accuracy against multi-step attacks).

Relation to Prior Work

Four inference DSE systems establish that the hardware search itself is solved well, and each supplies a technique we reuse:

  • MAESTRO describes a mapping with spatial and temporal primitives. Its analytical model predicts runtime and energy within 3.9% of cycle-accurate RTL, at about 170,000 designs per second.
  • AutoDNNchip prunes millions of points with a coarse predictor, then polishes the survivors with a fine-grained simulator.
  • AutoDSE steers FPGA pragma search by whichever resource is the bottleneck.
  • HIDA compiles hierarchical dataflow for high-level synthesis.

All four model only inference layers, so none can represent a backward pass to the input, weight-gradient work, or loops that repeat a minibatch. We extend MAESTRO's style of model rather than replace it. GANDSE uses a generative adversarial network to generate accelerator configurations. It borrows the word "adversarial" for a search method and never touches robustness, so it is a possible search engine for us, not a competitor.

The paper that motivates the proposal is the A³ accelerator for adversarial attacks. It observes that attack generation keeps weights fixed and needs under a tenth of the on-chip storage of CNN training. It turns the freed area into compute and reports 2–2.5× geometric-mean speedup over the memristor trainer Pipelayer at equal power. A³ is a single hand-designed memristor chip that accelerates only the inner loop. It leaves open what an end-to-end trainer should look like when attack passes and weight passes share one chip, and how that answer moves with $K$ or $m$. That open question is exactly our search.

Two algorithmic lines constrain the recipe axis rather than competing on it:

  • Fast is better than free shows single-step FGSM with random initialization reaches PGD-level robustness. It is sensitive to step size and schedule, and it already depends on FP16 arithmetic, which is why precision must be a joint variable.
  • Free AT reuses the input gradient from each weight backward pass by replaying minibatches. Its reported clean-accuracy drop as $m$ grows shapes our recipe grid. Its replay pattern is also a memory-traffic pattern that no hardware study has examined.

RobustBench supplies the evaluation discipline. It reports that original papers sometimes overestimated robustness by more than 10%, so every recipe is scored with AutoAttack.

Research Program

Step 1: the cost model. We first build a pass-typed analytical cost model. Each training step is decomposed into forward passes, input-gradient backward passes, and weight-gradient computations, each with explicit tensor lifetimes:

  • activations retained for the backward pass;
  • weight-gradient accumulators;
  • optimizer state;
  • for Free AT, the perturbation and minibatch held resident across $m$ replays.

A recipe compiles to a sequence of these passes. The model extends MAESTRO's per-layer reuse accounting to the extra matrix multiplies of backpropagation. It then adds a buffer-residency check that charges off-chip memory traffic when the working set of a pass exceeds on-chip capacity.

Step 2: validation. We validate the model against cycle-level simulation, and against a small number of FPGA builds generated through high-level synthesis, for a handful of layers and pass types. The target is error comparable to MAESTRO's reported inference error.

Step 3: the recipe grid. Separately, we measure the robustness side on GPU. The grid covers roughly forty configurations: PGD-$K$ with $K \in \{2,3,5,7,10\}$, Fast AT with and without random initialization over step sizes, Free AT with $m \in \{2,4,8\}$, and each at FP32, FP16, and emulated INT8. For every configuration we record clean accuracy, AutoAttack accuracy, and whether catastrophic overfitting occurred across seeds. This table defines which recipes are feasible for a given robustness target $r$.

Step 4: the joint search. For each feasible recipe, we search hardware configurations under an area budget. The search space covers processing-element array shape, buffer size and banking, and tiling. We use bottleneck-guided coordinate search in the manner of AutoDSE, with a coarse-then-fine pruning stage. The output is a Pareto front of energy against robust accuracy. We then ask how the optimal hardware moves as the recipe changes.

Baselines

Do-nothing reference. The hardware is the design that MAESTRO-style search finds optimal for forward inference of the same network under the same area budget. It runs each recipe with the best schedule our model can find for it. This is the "use an inference accelerator" answer. Beating it means reaching the same AutoAttack accuracy at meaningfully lower modeled energy. We pre-register 1.3× as the threshold that justifies a dedicated design.

Lifetime-blind DSE. This is the strongest competitor on the hardware axis, and it is the ablation that isolates our key ingredient. It uses the identical search and budget, but the cost model treats every pass as a standard training step with full weight-gradient work and full activation retention. This is what one gets by feeding the backward graph to an AutoDNNchip-style tool. If lifetime-aware search does not beat it at matched robustness, the pass-typing idea adds nothing.

Hand-designed A³-style point. We add a chip whose buffers are shrunk for attack passes and whose freed area is reallocated to compute, then run it end to end with weight passes included. A³ itself reports 2–2.5× geometric-mean speedup over Pipelayer at equal power. That figure covers attack generation on memristor hardware only, so it is a reference for what inner-loop specialization buys, not a directly comparable number.

Recipe baselines, reported numbers.

  • Fast is better than free reports 45% robust accuracy against PGD on CIFAR-10 in 6 minutes, versus 80 hours for PGD-10.
  • The same paper reports 43% on ImageNet in 12 hours.
  • Free AT reports about 46% against PGD-20 on CIFAR-10 in about 13 hours. The Fast AT paper quotes 10 hours for Free on CIFAR-10.
  • No reported result was found on Lacuna for AutoAttack accuracy of these recipes, so we measure it ourselves, with identical epochs, seeds, and tuning budget per recipe.

Recipes are tuned with the same budget of GPU trials before any hardware search begins, so no recipe gains from extra tuning.

Why not just…?

Why not just run Fast AT on a GPU, since it already takes minutes? For datacenter CIFAR-scale training this is a real threat. We do not claim to beat a GPU in wall-clock time there. The case for a dedicated design rests on regimes where energy and area dominate: repeated robust fine-tuning on devices, and ImageNet-scale runs, where Fast AT still takes 12 hours. A³'s observation, that general training hardware provisions storage attack passes do not use, suggests the waste is structural. We will report measured GPU energy per trained model next to modeled accelerator energy, and state plainly that the comparison crosses measured and modeled numbers. The claim that survives this caveat is the relative one: pass-aware designs versus inference-optimized and lifetime-blind designs, all under the same cost model.

Why not just give an existing DSE tool the backward graph? That is exactly the lifetime-blind baseline. The reason to expect it to fall short is that it charges attack passes for weight-gradient work and storage they do not need, and it cannot see Free AT's replay residency. If it matches our Pareto front within noise, we will report that the existing tools suffice.

Why not just pick one robust-training chip and swap recipes in software? If a single hardware configuration is near-optimal for every recipe, joint search is unnecessary. We test this directly by measuring the energy regret of each recipe's optimal chip when it runs the other recipes. Small regret everywhere would falsify the co-design claim.

Why not just use robustness tokens and skip full adversarial training? Robustness Tokens restores DINOv2-G accuracy under attack from 0.3% to 63.1% by tuning 10 tokens for 200 steps. However, it applies only to transformers and fails if the attacker learns the token weights. We treat it as a different regime, and include a token-tuning recipe on ViT workloads as an extension if time permits.

Proposed Experiments

Workloads. The main setting is PreAct ResNet-18 on CIFAR-10 at $\ell_\infty$, $\epsilon=8/255$, evaluated with AutoAttack through RobustBench's protocol. Robustness is measured on GPU. Energy and latency are modeled for area budgets that span edge-class to mid-size accelerators. ImageNet ResNet-50 is included for cost modeling only, using published recipe settings.

Decisive check 1: pass-typing matters. At matched AutoAttack accuracy, does lifetime-aware search find designs that dominate both the inference-optimized chip and lifetime-blind DSE on energy?

Decisive check 2: the optimum moves with the recipe. Does the optimal buffer-to-compute ratio shift monotonically as the recipe moves from Free AT (large resident working set) through PGD-$K$ to single-step Fast AT? Is the cross-recipe regret large enough to justify choosing hardware and recipe together?

Decisive check 3: precision and overfitting. Does the energy-optimal precision for Fast AT raise the rate of catastrophic overfitting across seeds? If so, the robustness constraint visibly changes which hardware is chosen, and that interaction is the strongest evidence that hardware search must see robustness.

Model-validity check. We compare modeled against simulated cost for sampled pass types, and report the error. Any headline conclusion that flips within that error band will be withdrawn.