Dynamic neural networks — models that skip layers, prune image tokens, or exit early when an input looks easy — save two to four times the compute of static networks on average, which is why they dominate battery-powered deployments (Dynamic Neural Networks: A Survey). But the same adaptivity creates a security hole: an adversary can add imperceptible perturbations that do not change what the model predicts, only how much it computes — a so-called energy-depletion or slowdown attack (SoK: Efficiency Robustness of Dynamic Deep Learning Systems). Each attack paper evaluates on its own models, its own metrics (FLOPs recovered, token utilization, latency), and its own hand-tuned hyperparameters, so nobody can say which dynamic architectures are actually safest or which defenses actually work. This is precisely the pathology that standardized benchmarks have fixed elsewhere: RobustBench showed that adversarial-robustness claims were inflated by more than 10% when defenses were tested against weak, custom attacks, and HarmBench showed that a single unstandardized generation parameter swings attack success rates by up to 30%.
We propose AutoSlow, an AutoAttack-style ensemble of parameter-free efficiency attacks with a fixed dual-metric scoring engine, and we will use it to re-adjudicate the published record of efficiency defenses. The core scientific bet is empirical and falsifiable: just as RobustBench exposed robustness overestimation in integrity defenses, a standardized worst-case efficiency attack will show that today's defenses — adversarial training, token caps, input smoothing — are weaker and more costly than their papers claim, because each was validated only against its authors' own attack under its own tuning.
Core Claim
No published efficiency defense has ever faced a strong, fairly-tuned, cross-mechanism attack suite. SlowFormer reports that adversarial training cuts its own patch attack from 100% to 34% success on A-ViT — but that number was obtained against one attack, tuned by the defense's authors. We claim that under a parameter-free ensemble that adapts its objective to the victim's dynamic mechanism (exit entropy, gate states, or token significance), the effective robustness of published defenses will drop materially, and the true "security tax" — the permanent clean-input compute penalty a defense pays — will turn out to be larger relative to the protection bought than the literature suggests. The deliverable is an executable benchmark: a containerized suite (NerfBaselines-style, since 12 of 19 published codebases in that field had already rotted) plus a versioned model zoo of dynamic networks spanning the D1 taxonomy of SoK: Efficiency Robustness of Dynamic Deep Learning Systems: dynamic depth, dynamic width, and token sparsification.
Research Program
The ensemble components already exist separately; the research is in unifying them under a parameter-free protocol and a single scoring rule. Each component attacks a different internal control signal: exit-head entropy maximization from A Panda? No, It's a Sloth: Slowdown Attacks on Adaptive Multi-Exit Neural Network Inference, Lipschitz-smoothed gate-opening with gradient projection from GradAuto, complexity gradient masking and the nonlinear power loss from GradMDM, token-significance uniformity via KL divergence and halting-score MSE from DeSparsify, universal patches from SlowFormer, and latent-entropy inflation for split-computing models from Resilience of Entropy Model in Distributed Neural Networks. AutoSlow runs them all with automatic step-size schedules in the spirit of APGD (the parameter-free core of RobustBench's AutoAttack) and reports the worst case, with every attacker held to the same perturbation budget and the same optimization-compute budget — a point the field has ignored even though A Panda? No, It's a Sloth itself concedes that crafting one perturbation costs the attacker more energy than the victim spends on it.
The scoring engine reports, for each model, a compute recovery ratio
$$\rho = \frac{C_{\text{attacked}} - C_{\text{clean}}}{C_{\max} - C_{\text{clean}}},$$
where $C$ is measured twice: once as FLOPs (comparable to prior work) and once as joules and wall-clock latency on a fixed edge device. This dual reporting directly tests the surrogate-versus-hardware disconnect that SlowFormer names as its own main limitation and that ILFO argues away by claiming intermediate activations are a good energy proxy — a claim nobody has checked across architectures, and which Dynamic Neural Networks: A Survey doubts because skipped layers do not translate linearly into saved joules on real accelerators. Defense evaluation is a four-axis Pareto report (clean accuracy, clean FLOPs, attacked FLOPs, attacked accuracy) with defense hyperparameters swept under a fixed budget, following the tuning-trap lesson of BARS: Towards Open Benchmarking for Recommender Systems, where 8,000 budgeted experiments revealed that properly tuned old baselines beat modern deep models.
Proposed Experiments
The model zoo covers early-exit and layer-skip networks (MSDNet, SkipNet, SACT), dynamic width (ManiDP), and token-sparsifying Vision Transformers (A-ViT, ATS, AdaViT) on CIFAR-10, CIFAR-100, Tiny ImageNet, and ImageNet, plus a split-computing pipeline for the entropy attack. The first study validates the ensemble: can parameter-free AutoSlow match the best hand-tuned single attacks on their home turf? The reference numbers to match are high — GradMDM reports an Average Recovery Percentage of 99.3% on ImageNet for SkipNet, and SlowFormer fully restores A-ViT's compute from 0.87 to 1.3 GFLOPs — so success on this axis means coming within a few points without any per-architecture tuning, while also working on mechanisms where those attacks have no reported results (GradMDM, for instance, has no reported result found on Lacuna for token-sparsifying models).
The second study is the headline: re-evaluating published defenses under AutoSlow with defense hyperparameters re-tuned under an equal budget. We re-run the adversarial training of SlowFormer, the hard token cap of DeSparsify, and the denoising defense of Resilience of Entropy Model in Distributed Neural Networks, each against the full ensemble including adaptive variants, since that paper's own authors showed their attacker shifts perturbation frequency to evade the denoiser. The third study is the hardware calibration: run the FLOP-recovery leaders on an edge module and a desktop GPU, and quantify how much of the algorithmic recovery survives as measured energy — DeSparsify found a 72% energy surge and 37% memory surge on an RTX 3090, but no reported result found on Lacuna for any efficiency attack measured in joules on edge hardware.
Baselines
The naive reference point is doing nothing: report the clean-input compute of each model and the compute under Gaussian noise. This matters because it quantifies how much of the "attack" effect is trivial — ILFO reports that random noise only increases workload by 3–4%, which is exactly the floor that makes the attack literature meaningful, and our benchmark should report it alongside every model so users can see vulnerability above noise. It will be run under identical data splits and identical FLOP counters, with no tuning.
The strongest existing competitor is GradMDM (GradMDM: Adversarial Attack on Dynamic Networks), the current record holder: 99.3% average recovery on ImageNet SkipNet, beating GradAuto's 88.9% on SkipNet+HRL, which in turn exposed ILFO's 46.9% as gradient-conflict-limited. It is the strongest because its complexity gradient masking and power loss directly solve the optimization pathologies that sank earlier attacks. To be fair, we run GradMDM with its published hyperparameters and also with a budget-matched hyperparameter sweep equal to the total tuning budget AutoSlow's components receive, on exactly our model zoo and splits. AutoSlow does not need to exceed 99.3% on SkipNet — on a single saturated target that would be meaningless. Beating GradMDM means matching its recovery within a few points on its home benchmark while achieving comparable recovery on at least two mechanisms where GradMDM has no reported result (token sparsification and split-computing entropy models), with zero per-model tuning. If GradMDM with tuning matches AutoSlow everywhere, the ensemble's value collapses to convenience, and we will say so.
The critical ablation isolates the parameter-free ingredient: an "AutoSlow-tuned" variant where each ensemble member's step sizes and loss weights are hand-tuned per architecture, versus the automatic-schedule version, plus a "max-of-attacks" run that simply takes the worst case over the original published attack scripts with their default settings. This ablation is what tests the RobustBench analogy: RobustBench found single hand-tuned attacks under- or over-perform unpredictably, and A Sober Look at the Unsupervised Learning of Disentangled Representations and their Evaluation found seeds and hyperparameters dominate architecture choices in a related setting — if default-setting max-of-attacks already reproduces the tuned ensemble's worst case everywhere, the parameter-free machinery is unnecessary and the benchmark can ship with the simpler protocol.
Why not just…?
Why not just use RobustBench and log inference latency? RobustBench cannot host these models: it explicitly requires static, deterministic architectures with no inference-time control flow, which every model in our zoo violates. More fundamentally, its attack objectives are wrong for this problem: A Panda? No, It's a Sloth showed that standard misclassification attacks often speed up dynamic networks, because confident wrong answers trigger early exits, and Dynamic Neural Network is All You Need: Understanding the Robustness of Dynamic Mechanisms in Neural Networks confirmed label-flipping attacks do not reliably slow them. Running integrity attacks and timing the model measures the wrong adversary.
Why not just compute the worst case analytically — the static network's FLOP count is a known upper bound? Empirically, attacks do not reach the bound uniformly, so the analytic number tells you nothing about vulnerability. SlowFormer restores 100% of compute on A-ViT but only 60% on ATS and 40% on AdaViT; the universal attack of A Panda? No, It's a Sloth achieves only a 5–45% efficacy reduction. And even at full FLOP recovery, Dynamic Neural Networks: A Survey notes FLOPs map nonlinearly to runtime and joules on real hardware, which is why our benchmark reports both. The analytic bound is not a competitor; it is the ceiling our $\rho$ score normalizes against.
Why not just harden models with simple heuristics — token caps, JPEG compression, threshold tightening — instead of building a benchmark? Because the heuristics are exactly the defenses whose claims we propose to test, and the existing evidence already shows they are costly and leaky: the token cap of DeSparsify damages accuracy on legitimately complex images; SlowFormer and A Panda? No, It's a Sloth both report that adversarial training permanently raises clean-input compute; and Resilience of Entropy Model in Distributed Neural Networks shows its denoiser is evaded by shifting perturbation frequency. Without a standardized worst-case attack, we cannot tell whether any heuristic is better than the others or than nothing — that comparison is the benchmark's product.
Why not just take the max over all published attack scripts with their default settings? This is the most serious alternative, and it is a real threat: it is cheaper, and our own ablation runs it. We expect it to fall short for two documented reasons — default settings were tuned per paper on per-architecture setups, and GradAuto showed attack performance collapses across architectures (ILFO drops from near-full recovery to 46.9% on SkipNet+HRL) — but if the ablation shows max-of-defaults matches the tuned ensemble on our zoo, we will adopt the simpler protocol and the contribution narrows honestly to the scoring engine, hardware calibration, and defense re-evaluation.
Decisive Checks
Three outcomes would decide the proposal's fate. First, if parameter-free AutoSlow cannot get within roughly five recovery points of the tuned best single attack on each mechanism, the RobustBench analogy fails and the benchmark should ship as a protocol rather than an ensemble. Second, if the published defenses hold up under the budget-matched ensemble — if the SlowFormer adversarial training keeps success near its reported 34% rather than climbing back toward 100% — our overestimation hypothesis is wrong, and the benchmark's value becomes the negative result that current defenses are adequate. Third, if the joule measurements on edge hardware track the FLOP-based $\rho$ scores closely, then ILFO's proxy argument is vindicated and future benchmarking can stay algorithmic; if they diverge, the field's headline numbers, including DeSparsify's desktop-GPU energy surge, have been calibrated on the wrong hardware all along. Any of the three is a publishable finding; the benchmark, its model zoo, and the defense Pareto data ship regardless.
References
- Dynamic Neural Networks: A Survey
- SoK: Efficiency Robustness of Dynamic Deep Learning Systems
- RobustBench: a standardized adversarial robustness benchmark
- HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal
- SlowFormer: Adversarial Attack on Compute and Energy Consumption of Efficient Vision Transformers
- NerfBaselines: Consistent and Reproducible Evaluation of Novel View Synthesis Methods
- A Panda? No, It's a Sloth: Slowdown Attacks on Adaptive Multi-Exit Neural Network Inference
- GradAuto: Energy-Oriented Attack on Dynamic Neural Networks
- GradMDM: Adversarial Attack on Dynamic Networks
- DeSparsify: Adversarial Attack Against Token Sparsification Mechanisms
- Resilience of Entropy Model in Distributed Neural Networks
- ILFO: Adversarial Attack on Adaptive Neural Networks
- BARS: Towards Open Benchmarking for Recommender Systems
- A Sober Look at the Unsupervised Learning of Disentangled Representations and their Evaluation
- Dynamic Neural Network is All You Need: Understanding the Robustness of Dynamic Mechanisms in Neural Networks