Abstract
As Vision Transformers (ViTs) are increasingly deployed on resource-constrained edge devices, adaptive computation techniques have emerged to improve energy efficiency by dynamically reducing token processing. This paper introduces SlowFormer, a novel universal adversarial patch designed to compromise the efficiency of these models. By applying a small, optimized patch to any input image, SlowFormer forces adaptive ViTs to perform maximum computation, effectively negating their energy-saving benefits. Our results demonstrate that a patch covering as little as 2-8% of the image can restore models to 100% computational capacity, posing a significant threat to the battery life and operational range of mobile and autonomous systems.
Introduction
The deployment of deep learning models on edge devices, such as delivery robots and drones, necessitates high energy efficiency. Adaptive Vision Transformers address this by dynamically "dropping" unimportant background tokens and focusing computation on relevant image regions. While adversarial research typically targets model accuracy, SlowFormer explores a vulnerability in efficiency. An adversary can craft an input that forces a model to operate at maximum capacity, leading to rapid battery depletion or thermal overheating without necessarily altering the final classification.
Methodology
SlowFormer is the first universal adversarial patch optimized to maximize the computation of efficient ViTs. Unlike previous energy attacks requiring full-image perturbations, SlowFormer is a static patch learned on a training set and applicable to any input at inference time.
The attack targets three primary efficient ViT frameworks:
- A-ViT: Employs a halting mechanism to terminate token processing early.
- ATS (Adaptive Token Sampling): Samples significant tokens while discarding others.
- AdaViT: Uses a decision network to skip patches, heads, or entire blocks.
Specific loss functions were developed to subvert the internal logic of these frameworks. For ATS, the patch is optimized to produce a uniform attention distribution, preventing the model from identifying and dropping "unimportant" tokens.
Experimental Setup
The effectiveness of SlowFormer was evaluated using two standard benchmarks:
- ImageNet-1K: A large-scale dataset comprising 1.3 million training images across 1,000 categories.
- CIFAR-10: A smaller dataset used to verify the attack's generalizability across different scales.
Analysis and Key Results
Our analysis reveals that the global self-attention mechanism of Transformers allows a tiny patch (as small as 2% of the image area) to influence the computational flow of the entire image. This makes the attack highly effective even when the patch is placed inconspicuously in a corner.
Key results include:
- Maximum Computation Restoration: On A-ViT (ViT-Tiny), SlowFormer successfully restored computation to its maximum level (1.3 GFLOPs), completely nullifying efficiency gains.
- Minimal Footprint: An 8% patch was sufficient for maximum attack success in most scenarios, while a 2% patch (32x32 pixels) achieved 90% success on A-ViT-Tiny.
- Stealth Potential: The attack can be tuned to maintain high classification accuracy while maximizing energy drain, making the attack difficult to detect through standard performance monitoring.
Conclusion and Future Work
SlowFormer demonstrates a critical security flaw in adaptive ViTs. While adversarial training can mitigate the attack's success (reducing it from 100% to 34% in some tests), it often comes at the cost of baseline efficiency and accuracy.
A limitation of this study is the use of FLOPs as a proxy for energy consumption. Future research should involve direct wattage measurements on physical hardware and testing in real-time robotic scenarios. This work underscores the need for "efficiency-robustness" as a standard metric for edge-deployed AI systems.
Summary updated