A standard image classifier can already flag unfamiliar inputs almost for free. The trick is to sum its raw output scores (logits) with a log-sum-exp and read the negative of that sum as an "energy". Familiar inputs get low energy and strange ones get high energy. This energy score cuts false alarms on out-of-distribution (OOD) data, meaning inputs from outside the training distribution, well below those of softmax confidence, and fine-tuning on a pool of auxiliary outlier images widens the gap further without costing clean accuracy (Energy-based Out-of-distribution Detection). Defending the same classifier against adversarial examples is far more expensive. Adversarial training runs a multi-step attack inside every training step. Even then, the best CIFAR-10 models give up more than ten points of clean accuracy and stay below roughly 67% robust accuracy (RobustBench). We ask whether a classifier can meet a much weaker requirement cheaply. For every small perturbation of a test image, it should either predict the right label or refuse to answer, and it should refuse by the same energy threshold it already uses to reject OOD inputs.
The obvious version of this idea fails, and that failure is the starting point. Energy is a differentiable function of the logits, so an attacker who knows about the detector can push energy down while flipping the label. This is exactly how Carlini and Wagner broke ten earlier detectors, at a cost of only about 10% more distortion (Adversarial Examples Are Not Easily Detected). Our proposal is to train against that attacker directly, using a single cheap attack step per batch and a loss that is satisfied whenever the perturbed point is either classified correctly or given high energy. The model is never forced to classify correctly inside the whole perturbation ball. We argue this relaxation is where both the clean-accuracy cost and the compute cost of adversarial training can be recovered.
Core Claim
A single-step, attack-aware energy objective can produce a classifier with an energy threshold at which three things hold together. First, clean accuracy stays within about two points of standard training. Second, accuracy with rejection under full adaptive attack, meaning the fraction of attacked test points that are either correct or rejected at a threshold accepting 95% of clean points, is competitive with rejection methods built on PGD adversarial training. Third, OOD detection is no worse than energy-bounded learning. All of this should cost two to three forward-backward passes per example rather than the eleven or so of PGD-10 training; that figure is simple step counting, not a measured result.
The mechanism is written most clearly as the loss on a perturbed training point. Let $f$ be the network, $E(x) = -\log \sum_j e^{f_j(x)}$ the energy, $y$ the label, and $m_{out}$ the energy margin above which inputs are rejected. The perturbation $\tilde{x} = x + \delta$ comes from one signed-gradient step of size $\epsilon$ on the attacker's joint objective $\mathrm{CE}(f(x+\delta), y) - \lambda E(x+\delta)$, where CE is cross-entropy and $\lambda$ trades label-flipping against energy suppression. The loss on that perturbed point is
$$ \ell_{\text{cor}}(\tilde{x}, y) = \min\Big( \mathrm{CE}(f(\tilde{x}), y),\; \beta \big[\max(0,\, m_{out} - E(\tilde{x}))\big]^2 \Big), $$
where $\beta$ puts the two terms on a common scale. The full objective adds $\ell_{\text{cor}}$ to the clean cross-entropy and to the two hinge terms of energy-bounded learning. The first hinge keeps clean data below an energy margin $m_{in}$, and the second keeps auxiliary outliers above $m_{out}$.
Two parts of this design are new relative to earlier work. The first is the soft minimum, which lets the model choose between the two acceptable outcomes point by point. The second is that the inner attack is the adaptive attack itself rather than a label-only attack. Near the class boundary, where the adversarial Bayes classifier must differ from the standard one (A Notion of Uniqueness for the Adversarial Bayes Classifier), the model can abstain instead of bending its decision boundary. That is the part of the robustness-accuracy trade-off we expect to recover.
Relation to Prior Work
Three papers supply the method. Energy-bounded learning (Energy-based Out-of-distribution Detection) supplies the hinge losses and the OOD evaluation protocol, but it is never tested against an attacker. Carlini and Wagner's joint loss supplies our inner attack. Tramèr et al.'s audit of 13 defenses, all of which were broken (On Adaptive Attacks to Adversarial Example Defenses), supplies the evaluation discipline. That discipline includes their warning (their theme T6) that extra defenses layered onto adversarial training can distort the training-time attack and quietly weaken the model.
The closest competitors connect rejection to robustness. Adversarial Training with Rectified Rejection trains an auxiliary head that predicts the probability of the true class, and it argues that an attacker cannot easily maximize that estimate and standard confidence together. It is cheap on top of its backbone, but it assumes a PGD adversarially trained backbone and a bounded confidence score, and it does not target OOD rejection. Stratified Adversarial Robustness with Rejection rejects inputs whose predictions are inconsistent within a small neighbourhood, which requires running PGD at test time on every input. Its framing constrains our metric: rejecting slightly perturbed clean points should count as a real cost, and we charge it. We differ from both by removing multi-step attacks from training and from inference, and by using one threshold for adversarial and OOD rejection.
Energy-based adversarial training motivates the work and also warns against parts of it. Your Classifier Can Do More (EB-JDAT) reaches 66.30% AutoAttack robustness at 90.39% clean accuracy on CIFAR-10. It does so by adding Langevin sampling, a slow iterative way of drawing samples from the energy model, on top of a full adversarial inner loop, which is the opposite of cheap. It also never checks OOD detection. Understanding Adversarial Training with Energy-based Models is the paper that most seems to contradict us. It shows that catastrophic overfitting in single-step training, where robustness suddenly collapses mid-training, is preceded by a spike in the energy gap between perturbed and clean points. Its regularizer (DER) therefore shrinks that gap, while our objective deliberately widens the gap for misclassified perturbed points.
We take this conflict seriously. Its most likely consequence is that the single training step stops finding real adversarial points, leaving a landscape that looks robust only because gradients mislead the attacker. The experiments below are built to detect this. The two objectives are also not strictly opposed. DER suppresses uncontrolled energy spikes on all perturbed points, whereas we raise energy only where the label has already flipped.
Finally, Adversarial Distributions Against Out-of-Distribution Detectors shows that detectors which look nearly perfect on static benchmarks collapse under generated worst-case outliers; one detector's AUROC falls from 0.935 to 0.005 on SVHN. It also reports that detectors adversarially trained on perturbed outliers (ACET, GOOD, CEDA) fail to transfer to geometric transformations. That is a direct warning that our method might learn a brittle rule tied to the attack geometry, so we evaluate beyond it.
Research Program
The build is a WideResNet trained on CIFAR-10 and CIFAR-100 with the objective above, using the auxiliary outlier set protocol of energy-bounded learning, $\ell_\infty$ radius $8/255$, and one joint-objective step per batch. $\lambda$ is randomized per batch so the model does not overfit to a single mix of label-flipping and energy suppression.
The analysis has three parts. First, we check whether the soft minimum actually splits perturbed points into a "correct" group and a "rejected" group, and whether the rejected ones sit near the clean decision boundary, as the adversarial Bayes argument predicts. Second, throughout training, we track the gap between how often the single step succeeds and how often a 50-step joint attack succeeds. A widening gap is the signature of catastrophic overfitting and gradient masking. If it appears, we add a DER-style penalty restricted to correctly classified perturbed points and test whether that restores agreement between the two attacks. Third, we measure whether widening the energy range on perturbed points shifts clean ID energies enough to hurt OOD detection, especially on near-OOD splits such as CIFAR-10 versus CIFAR-100, where logit distributions overlap.
Baselines
The do-nothing reference is a standard WideResNet with a post-hoc energy threshold chosen to accept 95% of clean test points, attacked with the joint objective. We expect accuracy with rejection near zero, which would reproduce the Carlini–Wagner result for energy scoring. Carlini and Wagner never tested free-energy scoring directly, so this check is also a small contribution. Any configuration of our attack that fails to break this baseline is too weak and will be discarded.
The strongest existing method is Rectified Rejection on a PGD adversarially trained backbone. It is the most direct competitor because it already achieves cheap-at-inference rejection with adaptive evaluation. The file reports about 58% accuracy at TPR-95 under PGD-100 on CIFAR-10 (Adversarial Training with Rectified Rejection). We will retrain it with the same architecture, epochs, and augmentation, and evaluate it under our joint adaptive attacks against its own rejection score as well as under PGD-100. Beating it means matching its accuracy with rejection within two points while using clearly fewer training passes, or exceeding it at equal passes, with at least five points higher clean accuracy in either case. We will also report EB-JDAT's published 66.30% AutoAttack robustness and 90.39% clean accuracy as a ceiling for no-rejection robustness. EB-JDAT is not a like-for-like rejection method, and we found no training-cost figure for it on Lacuna.
Two further references complete the picture. The first is a RobustBench PGD-trained model with a post-hoc energy threshold, and the second is a single-step DER model with the same threshold (Understanding Adversarial Training with Energy-based Models). Both receive the same threshold rule and the same attacks. The key ablation replaces the joint-objective inner step with a plain cross-entropy step, which is essentially energy-bounded learning with FGSM points treated as outliers. This isolates whether attack-awareness during training is what matters. A second ablation replaces the soft minimum with a sum of the two terms, which forces both correct classification and high energy and should behave more like ordinary adversarial training. All methods share one tuning budget, set as a fixed number of full training runs per method, and we report GPU-hours alongside accuracy.
Why not just…?
Why not just threshold energy on an existing adversarially trained model? This is the strongest threat. Pretrained models are available from RobustBench, and their smoother energy landscape might already reject most successful attacks. However, it inherits the full PGD training cost and the clean-accuracy drop into the low 80s that RobustBench documents. RobustBench also reports that such models are underconfident, which suggests their energy scores may be poorly separated for OOD rejection. If this baseline matches our accuracy with rejection and our OOD numbers, our contribution shrinks to a compute saving. If it also matches our clean accuracy, the idea has failed.
Why not just use Rectified Rejection? It is efficient at inference and already evaluated adaptively. Its limitation for this question is that it needs a PGD-trained backbone and a bounded confidence score, and it was not designed for OOD inputs. The comparison above measures directly whether dropping the PGD backbone costs robustness.
Why not just add adversarial examples to the outlier set of energy-bounded learning? This is our first ablation. The ACET/GOOD results in Adversarial Distributions Against Out-of-Distribution Detectors suggest that training against one attack form produces rigid rules, and Carlini–Wagner suggests an attacker who targets energy will find points the label-only attack never produced. If this ablation survives the joint adaptive attack as well as the full method does, the soft minimum and attack-aware step are unnecessary, and we will report that.
Proposed Experiments
We evaluate on CIFAR-10 and CIFAR-100 with SVHN as a far-OOD set, plus the OpenOOD near-OOD and far-OOD splits (OpenOOD). For those splits the file's reference point is PixMix at 93.1% AUROC on CIFAR-10 near-OOD. For the energy-bounded baseline, the reported reference is 3.32% average FPR at 95% TPR on CIFAR-10.
Adversarial evaluation follows the Tramèr et al. checklist. We attack the full pipeline end to end with multi-step APGD on the joint objective, sweeping $\lambda$ and restarts. We run a threshold-aware attack that minimizes energy only until it falls below the threshold and then maximizes the loss. We also run gradient-free Square Attack, so that success is not an artifact of misleading gradients, and standard AutoAttack for comparability. For OOD robustness, we perturb SVHN and OpenOOD outliers to lower their energy under the same budget and report AUROC. Because both Stochastic Security and our joint attack show that purely pixel-level tests can mislead, we add rotations and translations as an unseen threat to probe the transfer failure seen in earlier OOD adversarial training. We report CIFAR-10-C clean acceptance as well, since rejection-happy behaviour on corrupted clean images is the failure CCAT displayed (Stratified Adversarial Robustness with Rejection).
The decisive checks are four. The do-nothing baseline must collapse under our attack. The single-step versus 50-step success gap during training must stay small; otherwise any robustness we report is masking. The full method must beat the non-adaptive ablation under adaptive evaluation by a clear margin. At a matched clean-acceptance rate of 95%, the full method must deliver accuracy with rejection comparable to Rectified Rejection with markedly higher clean accuracy and lower training cost, while keeping near-OOD FPR at 95% TPR no worse than energy-bounded learning. Failing the second check falsifies the approach outright. Failing only the last check leaves a narrower but still useful finding: attack-aware energy training gives cheap adversarial rejection but costs OOD detection.
References
- Energy-based Out-of-distribution Detection
- RobustBench: a standardized adversarial robustness benchmark
- Adversarial Examples Are Not Easily Detected: Bypassing Ten Detection Methods
- A Notion of Uniqueness for the Adversarial Bayes Classifier
- On Adaptive Attacks to Adversarial Example Defenses
- Adversarial Training with Rectified Rejection
- Stratified Adversarial Robustness with Rejection
- Your Classifier Can Do More: Towards Bridging the Gaps in Classification, Robustness, and Generation
- Understanding Adversarial Training with Energy-based Models
- Adversarial Distributions Against Out-of-Distribution Detectors
- OpenOOD: Benchmarking Generalized Out-of-Distribution Detection
- Stochastic Security: Adversarial Defense Using Long-Run Dynamics of Energy-Based Models