Abstract
As deep learning models transition to edge devices with limited power budgets, adaptive computation has emerged as a key strategy for energy efficiency. This paper introduces SlowFormer, a novel adversarial attack that targets the "utility bill" of adaptive Vision Transformers (ViTs). By utilizing a small, universal patch, SlowFormer tricks models into abandoning their power-saving features, forcing them to run at maximum computational cost. Our results demonstrate that this "energy-depletion" attack can effectively drain the battery of mobile and autonomous systems by restoring up to 100% of the baseline computational overhead.
Introduction
Energy efficiency is paramount for deep learning applications on edge devices such as drones, AR glasses, and autonomous robots. To optimize power consumption, researchers have developed adaptive computation methods. These models dynamically allocate computational resources based on input complexity; for instance, a self-driving system might "drop" redundant image patches (tokens) when processing a simple scene to save Floating Point Operations (FLOPs).
While traditional adversarial attacks focus on inducing misclassification, SlowFormer targets the model's operational efficiency. We identify a critical vulnerability where an attacker can manipulate an input to make a simple scene appear complex, forcing the model to utilize its maximum power. For battery-constrained hardware, this can significantly reduce operational range or cause overheating.
Methodology
SlowFormer is a universal adversarial patch attack. Unlike previous "slowdown" attacks that required unique pixel-level perturbations for every input, SlowFormer generates a single, static patch that can be applied to any image to trigger maximum computation.
The attack targets three state-of-the-art efficient ViT frameworks:
- A-ViT: Employs a "halting" mechanism to terminate token processing early.
- ATS (Adaptive Token Sampling): Samples only the most salient tokens at each transformer layer.
- AdaViT: Uses a decision network to skip patches, attention heads, or entire blocks.
The Optimization Strategy
Instead of maximizing classification loss, SlowFormer optimizes the patch to maximize Usage Loss. The goal is to deceive the model's internal decision-making components into believing that every part of the image is essential, thereby preventing the model from pruning or skipping any computation.
Experimental Results
We evaluated SlowFormer on the ImageNet-1K and CIFAR-10 benchmarks. The attack proved highly effective across different model architectures:
- A-ViT (ViT-Tiny): Achieved a 100% success rate, raising compute from 0.87 GFLOPs back to the 1.3 GFLOPs baseline.
- ATS (ViT-Small): Increased computation by 60%, moving from 3.1 GFLOPs to 4.0 GFLOPs.
- AdaViT: Restored 40% of the computational overhead.
Qualitative analysis (see Figure 4) shows that while the efficient models naturally prune a large percentage of background tokens, the presence of the SlowFormer patch causes the model to retain nearly all tokens, effectively nullifying the benefits of adaptive computation.
Key Insights and Discussion
Global Attention Vulnerability
Because Vision Transformers utilize global self-attention, a small patch in a single corner can influence the processing of the entire image. The patch acts as a "distractor" that forces the model to maintain tokens even in distant, unrelated parts of the frame. We found that a patch occupying only 8% of the image area was sufficient to completely undo the efficiency gains of models like A-ViT.
Accuracy-Agnostic Attacks
SlowFormer is accuracy-agnostic. An attacker can choose to either destroy the model's accuracy while slowing it down or maintain high accuracy so the victim remains unaware of the attack while their battery drains.
The Security Tax
Testing standard adversarial training as a defense revealed a "security tax." While it reduced attack success (e.g., from 100% to 34% on A-ViT), it also increased the baseline computation cost of the model, making the "defended" model less efficient even when not under attack.
Limitations and Future Work
This study uses FLOPs as a proxy for energy consumption; future work should involve direct wattage measurements on physical hardware like NVIDIA Jetson modules. Additionally, while this work assumes a white-box scenario, exploring black-box attacks and extending this research to Large Language Models (LLMs) with adaptive compute remains a priority for securing the next generation of efficient AI.
Summary updated