Most adversarial attacks are "skin deep." By optimizing perturbations to flip a model's final classification label, attackers often exploit idiosyncratic "bugs" or high-frequency noise patterns specific to a single model's weights. While these attacks are devastatingly effective against the model they were trained on, they often fail to transfer to other, unknown "black-box" models.
To solve this, a growing body of research suggests moving the battleground from the final output layer to the network's internal representations. The core intuition is simple: while different models (like a ResNet and a Vision Transformer) have different decision boundaries, they must all extract similar semantic features—like the shape of a wheel or the texture of fur—to recognize an object. By disrupting these shared intermediate features, attackers can create "universal" perturbations that fool almost any architecture.
Disrupting the Internal Representation
Standard attacks focus on the loss function at the very end of the network. In contrast, the Feature Disruptive Attack (FDA), introduced by Aditya Ganeshan et al. in FDA: Feature Disruptive Attack, aims to systematically corrupt features at every layer. This effectively "blinds" the model to the image content entirely, making the attack potent across diverse tasks like image captioning and style transfer.
Other methods focus on specific types of feature manipulation to enhance transferability:
- Activation Matching: Nathan Inkawhich and colleagues developed the Activation Attack (AA) in Feature Space Perturbations Yield More Transferable Adversarial Examples. Instead of maximizing classification error, they force the source image's internal activations to match those of a completely different target image.
- Dispersion Reduction: In Enhancing Cross-task Black-Box Transferability of Adversarial Examples with Dispersion Reduction, Yantao Lu et al. proposed Dispersion Reduction (DR). This attack minimizes the "contrast" or variance of internal feature maps, making the image appear "featureless" to the AI.
- Feature Importance: Zhibo Wang et al. introduced the Feature Importance-aware Attack (FIA) in Feature Importance-aware Transferable Adversarial Attacks, which uses aggregate gradients to identify and disrupt the specific neurons that any model would likely find important for recognition.
Improving the Gradient Path
A major hurdle in intermediate attacks is the "vanishing" or "shattered" gradient problem caused by non-linear activations like ReLUs. If the gradient signal becomes too noisy as it backpropagates through the network, the resulting perturbation will be overfitted to the source model's specific non-linearities.
To fix this, Steven Y. Guo and Qizhang Li proposed Linear Backpropagation (LinBP) in Backpropagating Linearly Improves Transferability of Adversarial Examples. By treating the network as more linear during the backward pass (skipping the ReLU constraints), they generate smoother, more transferable gradients. Similarly, Jianping Zhang et al. used Neuron Attribution-based Attacks (NAA) in Improving Adversarial Transferability via Neuron Attribution-based Attacks to more accurately credit which internal features drive a decision, leading to more effective black-box perturbations.
The Layer Selection Problem
Despite the success of these methods, they all share a common Achilles' heel: sensitivity to layer selection.
As noted across nearly all the surveyed literature—from Yet Another Intermediate-Level Attack to the recent Pixel2Feature Attack (P2FA) by Renpu Liu—the effectiveness of a feature-space attack depends heavily on which layer is targeted.
- Early layers capture low-level textures that might be too model-specific.
- Late layers are too closely tied to the final classification logic, leading back to the overfitting problem.
- The "Sweet Spot" varies by architecture. A layer that works for a ResNet-50 might be suboptimal for an Inception-v3 or a Vision Transformer.
Currently, identifying the optimal layer remains a manual, empirical process. Researchers like Jihan Yang in An Adversarial Perturbation Oriented Domain Adaptation Approach for Semantic Segmentation and Huanrui Yang in DVERGE highlight that while we know intermediate features are the key to transferability, we lack a universal, architecture-agnostic heuristic for selecting them.
Beyond Attacks: Robustness and Generation
The principle of manipulating intermediate features isn't just for attackers; it's also a tool for defense and generative modeling.
In Robust SAM, Jiahuan Long et al. use Singular Value Decomposition (SVD) on specific intermediate layers to surgically harden the Segment Anything Model (SAM) against attacks. Meanwhile, in the realm of generative AI, Javad Rajabi et al. introduced Token Perturbation Guidance (TPG) in Token Perturbation Guidance for Diffusion Models, showing that shuffling intermediate token representations can improve the output quality of diffusion models without any retraining.
Whether the goal is to break a model, defend it, or improve its creativity, the consensus is clear: the most meaningful changes happen in the "hidden" layers where the model's true understanding resides.