Abstract
Deep neural networks have been shown to be fooled rather easily using adversarial attack algorithms. Practical methods such as adversarial patches have been shown to be extremely effective in causing misclassification. However, these patches are highlighted using standard network interpretation algorithms, thus revealing the identity of the adversary. We show that it is possible to create adversarial patches which not only fool the prediction, but also change what we interpret regarding the cause of the prediction. Moreover, we introduce our attack as a controlled setting to measure the accuracy of interpretation algorithms. We show this using extensive experiments for Grad-CAM interpretation that transfers to occluding patch interpretation as well. We believe our algorithms can facilitate developing more robust network interpretation tools that truly explain the network's underlying decision making process.
Introduction: The Vulnerability of Explainable AI
As deep learning moves into high-stakes fields like healthcare and autonomous driving, Explainable AI (XAI) has become a critical safety net. Tools like Grad-CAM are used to generate "heatmaps" that highlight which parts of an image led to a specific classification. For example, if a model identifies a "Soccer Ball," Grad-CAM should highlight the ball.
Previously, researchers found that "adversarial patches"—small, noisy stickers placed on an image—could easily force a misclassification. However, these patches were "loud" in interpretation maps; Grad-CAM would highlight the patch itself, effectively pointing a finger at the source of the error. This paper addresses a significant security gap: Can an adversary fool the classifier while simultaneously hiding their tracks from the interpretation tools?
Methodology: Dual-Objective Optimization
The authors build upon the standard adversarial patch framework. Normally, an attacker optimizes a patch ($z$) to minimize the cross-entropy loss for a target category ($t$). The innovation in this work is the introduction of a dual-objective loss function. They optimize for two things at once:
- Misclassification: Minimize the standard cross-entropy loss so the model predicts the target category.
- Interpretation Suppression: Minimize the "energy" of the interpretation heatmap (Grad-CAM) specifically within the area of the patch.
Mathematically, they add a regularization term $\lambda \sum_{ij} (\hat{G}^t(\tilde{x}) \odot m)$, where $\hat{G}^t$ is the Grad-CAM heatmap and $m$ is the binary mask of the patch location. By tuning the hyperparameter $\lambda$, they can balance how well the attack works versus how well it is hidden.
Key Results and Insights
The study utilized the ImageNet (ILSVRC2012) and PASCAL VOC-2012 benchmarks to test the attack across various architectures, including VGG19-BN, ResNet-34, and DenseNet-121.
Stealth and Success Rates
On VGG19-BN using ImageNet, a standard adversarial patch has ~76% of its interpretation "energy" concentrated on the patch. The proposed "Our Patch" reduced this to just 6.8%, making the patch nearly invisible in the heatmap. Despite the added constraint of hiding the patch, the attacks remained highly effective. On ResNet-34 and DenseNet-121, the hidden patches achieved over 90% target accuracy while maintaining very low energy ratios (8.2% and 5.3% respectively).
Cross-Method Transferability
Even though the attack is optimized specifically using Grad-CAM's gradients, the resulting patches also successfully fool "Occluding Patch" interpretation. This suggests the attack is manipulating something fundamental about how the network processes the image, rather than just exploiting a quirk of one specific visualization tool.
Conclusion
The most important takeaway is that current interpretation algorithms are not robust. They can be manipulated just as easily as the underlying classifiers. In many cases, the attack doesn't just hide the patch; it forces the interpretation algorithm to highlight features of the original object as the "cause" for the new (wrong) label.
The authors suggest that their attack should be used as a benchmark for future XAI research. A truly robust interpretation algorithm should be able to highlight the adversarial patch as the cause of a misclassification, even when the patch is trying to hide.
Summary updated