Deploying a neural network that is both small and hard to fool usually means paying for adversarial training many times over. Adversarial training teaches a model on deliberately perturbed inputs. Each update needs an inner search for the worst perturbation, which makes it roughly 10 to 40 times more expensive than ordinary training. Quantization stores weights in 8, 4, 2 or 1 bits instead of 32. It interacts badly with that search: small input perturbations can grow from layer to layer until they push activations into the wrong rounding bucket. Methods that fix this train a separate robust model for each bit-width. Practitioners also want to choose how much clean accuracy to give up for robustness, so they train one model per bit-width per trade-off setting. A deployment grid of four bit-widths and five trade-off points therefore costs twenty adversarial training runs.
We propose to replace that grid with a single run. A small conditioning network, a "generator," reads a requested configuration and outputs weight corrections for one shared backbone. The configuration has three parts: the bit-width, the attack budget, and a knob setting how much clean accuracy to trade for robustness. The corrections are merged into the backbone before rounding, so each deployed model is an ordinary integer network at the requested precision. During training, every minibatch samples a different configuration and runs the adversarial search for that configuration only. The expensive inner loop is therefore shared across the whole grid rather than repeated for each point. The claim is that a K-point grid costs a small multiple of one adversarial training run. We will test whether each generated model matches a model trained separately for that point, and whether this still holds when both are measured with attacks that are not fooled by the flat, step-like input landscape of quantized networks.
Core Claim
Robust features are largely shared across precisions and trade-off settings. What differs between operating points is a low-dimensional correction. It absorbs the error growth caused by rounding at a given bit-width, and it moves the model along the clean-versus-robust curve. If this is true, three things follow. First, a configuration-conditioned generator trained once with adversarial loss should come within about one point of robust accuracy of per-configuration adversarial quantization-aware training. Second, it should do so at a total cost of about two single runs instead of K. Third, the trade-off knob, the brittle mixing weight that currently has to be retuned for each budget, becomes an input chosen at deployment time.
This idea draws on three bodies of work, each in a different role. Defensive Quantization and Error-Silenced Quantization motivate it. Defensive Quantization found the error amplification effect and fixed it with an orthonormality penalty on weight matrices. Error-Silenced Quantization fixed it at 1 and 2 bits by pairing clean and adversarial internal activations. Both show that robust low-bit models are attainable, but both need a full min-max training run per bit-width.
CoA-LoRA supplies the method. It uses a two-layer MLP that maps a quantization configuration to adjustments of low-rank adapters (small added weight matrices), trained once and then adapted in milliseconds. It targets only clean accuracy on language tasks and has no threat model. We reuse its mechanism, condition it on robustness as well as precision, and train it under an adversarial objective.
Several papers constrain the design. The weight-generation line generates continuous full-precision weights from pre-trained model collections: Neural Network Diffusion, DeepWeightFlow, SANE and Parameter Manifold Purification. Parameter Manifold Purification denoises backdoored or noisy-label weights but never addresses test-time ℓ∞ attacks or quantization. Using any of these for our goal would require first adversarially training a whole collection of models, which is the very cost we want to avoid. We therefore deliberately do not build a zoo-trained diffusion model; we explain why in the section on alternatives. Finally, Pareto-Conditioned Diffusion conditions generation on points along a trade-off curve, but outside weight spaces. We borrow only its framing of sampling at a requested trade-off point.
Research Program
The backbone is a Wide ResNet, as used by Defensive Quantization. It keeps latent full-precision weights $W$. The generator $g_\phi$ takes as input a configuration $c = (b, \epsilon, \beta)$, where $b$ is bit-width, $\epsilon$ is the ℓ∞ attack budget, and $\beta$ is the clean-versus-robust weight. For each layer it outputs a low-rank correction $A_c B_c$ and quantizer clipping scales. The deployed layer is $Q_b(W + A_c B_c)$, where $Q_b$ rounds to $b$ bits. At inference time nothing remains of the generator; the result is a fixed $b$-bit network.
Training minimizes, over configurations sampled from the grid,
$$\mathbb{E}_{c}\Big[\mathcal{L}_{CE}(f_c(x),y) + \beta\,\max_{\|\delta\|_\infty\le\epsilon}\mathrm{KL}\big(f_c(x)\,\|\,f_c(x+\delta)\big)\Big].$$
Here $f_c$ is the quantized network at configuration $c$ and the inner maximization is a PGD search. The objective has the familiar clean-plus-robust-KL form. Gradients pass through rounding with the straight-through estimator, which treats rounding as the identity during the backward pass. Each step pays for one inner search at one configuration, so the cost per step equals one ordinary adversarial quantization-aware run.
We will also test two additions. One is the orthonormality penalty from Defensive Quantization applied to $W$. The other is the activation-pairing loss from Error-Silenced Quantization, applied only when $b \le 2$. Defensive Quantization reports that its penalty breaks down at those precisions.
Batch normalization statistics depend on the configuration. Parameter generators often fail without recomputing them. We therefore store one set of batch normalization statistics per grid point, which costs negligible memory. For configurations not seen in training, we recompute them with one clean pass over 1,000 images.
The program also includes an analysis question. Is the generator's correction genuinely low-rank and smooth in $b$? The file reports that robustness in quantized networks is non-monotonic in bit-width, so interpolating to an unseen bit-width may fail even where interpolating in $\beta$ succeeds. We will measure both and report either result.
Baselines
The naive reference point is post-training quantization of a public adversarially trained full-precision CIFAR-10 model from RobustBench, taken to each bit-width with no retraining. Its training cost at deployment time is zero. It is the right floor because it is what practitioners do today.
The strongest competitor on quality is per-configuration adversarial quantization-aware training. At 4 and 5 bits this is Defensive Quantization. Its paper reports 51.8% accuracy under attack for a 5-bit Wide ResNet on CIFAR-10, against 30.2% for a vanilla 5-bit model and 39.3% at full precision. At 1 and 2 bits the competitor is Error-Silenced Quantization, which reports 41.62% PGD accuracy for a 1-bit model against 47.06% at full precision. Both figures come from PGD evaluation. RobustBench found that papers often overestimate robustness by more than 10 points, so we will retrain both methods ourselves and evaluate them under our protocol. We will not compare against the published numbers. Each per-configuration model gets the same backbone, the same data, the same number of PGD steps, and a tuning budget for learning rate and $\beta$ equal to the generator's.
The strongest competitor on cost is a robust version of CoA-LoRA's Shared-LoRA baseline: one adversarially trained adapter reused at every bit-width and $\beta$. The key ablation isolates conditioning. Holding total training compute fixed, we compare the generator conditioned on $(b, \epsilon, \beta)$ against three variants: conditioning on $b$ only, conditioning on $\beta$ only, and an unconditioned shared correction.
We count a result as beating the baselines only if three conditions hold on a 20-point grid. The generated models must be within 1 point of robust accuracy of per-configuration training at every point. They must strictly exceed post-training quantization and Shared-LoRA. And the whole grid must use less than one quarter of the combined GPU-hours of the 20 separate runs.
Why not just…?
Why not just quantize a robust full-precision model after training? Defensive Quantization's 30.2% vanilla 5-bit result shows that error amplification erodes robustness under naive rounding. Error-Silenced Quantization reports that unassisted 1- and 2-bit models fail under PGD. This baseline is still in our grid. If post-training quantization lands within a point at 8 bits, we will say so and restrict our claims to lower precisions.
Why not just train each configuration separately? It is only K runs. This is the honest competitor, and it sets the quality ceiling. The proposal is worth doing only if amortization is real. The decisive plot is robust accuracy against cumulative GPU-hours for both strategies as K grows from 1 to 20. If per-configuration training reaches the grid more cheaply, for example because short fine-tunes from a shared robust checkpoint suffice, then the generator has no reason to exist. We include those short fine-tunes as a variant of this baseline so that this outcome can be seen.
Why not just interpolate between a clean and a robust model that share a starting point? The file's argument against model soups rests on permutation mismatch between independently trained models. That argument does not apply to two models fine-tuned from the same initialization, so this alternative is a real threat along the $\beta$ axis. We will run it at every bit-width: interpolate, then quantize. We expect it to cover the trade-off axis without absorbing the bit-width-specific rounding error. If it matches the generator, we drop $\beta$ from the conditioning.
Why not just train a diffusion or flow model on a zoo of robust checkpoints? The zoo would need dozens of adversarially trained quantized models. That costs more than the grid it is meant to replace. Such generators also produce continuous weights, whose fine structure rounding may destroy. At small scale we will train DeepWeightFlow on the per-configuration models we train anyway. This is a sanity check on whether zoo-based generation adds anything once the zoo already exists. It is not offered as a fair cost comparison.
Why not just use robustness tokens on a frozen backbone? Robustness Tokens recovers DINOv2-G from 0.3% to 63.1% ImageNet accuracy under white-box attack while keeping clean accuracy at 85.2%. It is a strong full-precision method. However, it relies on float attention in giant vision transformers, must be optimized separately for each model, and loses its protection if the tokens leak. None of this fits small integer networks at the edge. We treat it as a reference in the optional ImageNet stage rather than as a competitor on CIFAR-10.
Proposed Experiments
The core study is CIFAR-10 with a Wide ResNet over a 20-point grid. The bit-widths are 1, 2, 4, 5 and 8. The $\beta$ values are four points spanning mostly-clean to mostly-robust training. The attack budget is fixed at 8/255 ℓ∞, with a second sweep that adds 4/255. We report clean accuracy, robust accuracy, and GPU-hours at each point, and the area of the dominated region under the clean-versus-robust curve at each bit-width.
Replication tests generality: SVHN, which Defensive Quantization also used, and Tiny-ImageNet at 4 and 8 bits. An optional stage applies 8-bit quantization to a ViT-B on ImageNet, with Robustness Tokens as the reference.
Evaluation guards against false robustness from quantization, whose flat landscape masks gradients. Every model faces AutoAttack, which includes Square, an attack that needs no gradients. It also faces PGD computed through the straight-through estimator on the quantized network, and transfer attacks crafted on the full-precision latent backbone. The reported robust accuracy is the minimum over all of these. A model whose PGD accuracy exceeds its Square accuracy is flagged as masking gradients.
Three decisive checks settle the claim. The first is the cost-versus-quality curve against separate training, including the short fine-tune variant. The second is the conditioning ablation at fixed compute. The third holds out 3-bit and 6-bit, and one $\beta$ value, to see whether the generator generalizes to unseen settings. We expect the held-out $\beta$ to succeed. Failure at the held-out bit-widths would be direct evidence that non-monotonicity limits smooth conditioning on precision.
References
- Defensive Quantization: When Efficiency Meets Robustness
- Error-Silenced Quantization: Bridging Robustness and Compactness
- On-the-Fly Adaptation to Quantization: Configuration-Aware LoRA for Efficient Fine-Tuning of Quantized LLMs
- Neural Network Diffusion
- DeepWeightFlow: Re-Basined Flow Matching for Generating Neural Network Weights
- Towards Scalable and Versatile Weight Space Learning
- Parameter Manifold Purification
- Pareto-Conditioned Diffusion Models for Offline Multi-Objective Optimization
- RobustBench: a standardized adversarial robustness benchmark
- Robustness Tokens: Towards Adversarial Robustness of Transformers