Abstract
We present an adversarial framework to craft perturbations that mislead classifiers by accounting for the image content and the semantics of the labels. The proposed framework combines a structure loss and a semantic adversarial loss in a multi-task objective function to train a fully convolutional neural network. The structure loss helps generate perturbations whose type and magnitude are defined by a target image processing filter. The semantic adversarial loss considers groups of (semantic) labels to craft perturbations that prevent the filtered image from being classified with a label in the same group. We validate our framework with three different target filters, namely detail enhancement, log transformation and gamma correction filters; and evaluate the adversarially filtered images against three classifiers, ResNet50, ResNet18 and AlexNet, pre-trained on ImageNet. We show that the proposed framework generates filtered images with a high success rate, robustness, and transferability to unseen classifiers. We also discuss objective and subjective evaluations of the adversarial perturbations.
Introduction
Deep Neural Networks (DNNs) are notoriously fragile to adversarial perturbations. Traditional attacks typically fall into two categories: norm-bounded attacks, which introduce nearly invisible pixel fluctuations, and content-based attacks, which modify colors or textures more aggressively. However, these methods often suffer from two major limitations. First, norm-bounded noise can appear as unnatural "static," while content-based attacks may produce unrealistic colors. Second, most attacks aim for any categorical misclassification, which often results in semantically similar errors—for instance, misidentifying one dog breed for another rather than identifying a dog as a vehicle.
FilterFool is a novel framework designed to overcome these limitations by creating "unrestricted" perturbations that mimic legitimate image processing filters. By making adversarial images look like they have simply undergone standard processing—such as gamma correction or detail enhancement—the perturbations become semantically transparent to humans while forcing the model into semantically significant misclassifications.
Methodology
The FilterFool framework centers on a Fully Convolutional Neural Network (FCNN) trained to generate a perturbation ($\delta$) that, when added to an input image, mimics the effect of a target filter while achieving an adversarial goal. The training process is guided by a multi-task objective function consisting of two primary components:
- Structure Loss ($L_{Str}$): This loss ensures the generated image maintains the visual characteristics of the target filter (e.g., Log transformation, Gamma correction, or Detail Enhancement). It utilizes a combination of $L_2$ error and the Structural Similarity (SSIM) index to preserve edges and structural integrity.
- Semantic Adversarial Loss ($L_{S-Adv}$): To ensure misclassifications are semantically meaningful, the authors leverage WordNet to group the 1000 ImageNet classes into 11 broad semantic categories (e.g., dogs, birds, vehicles). The loss function specifically penalizes the model for predicting any label within the original semantic group, rewarding it for shifting the prediction to a different category.
Results and Discussion
Evaluation on the ImageNet-1K dataset demonstrates the high effectiveness of the FilterFool approach. The framework achieved a 100% categorical success rate on ResNet50 for several filter types. More importantly, the semantic success rate—the ability to force the model into a different broad category—remained between 95% and 100%.
Key findings include:
- Transferability: Attacks crafted for ResNet50 were highly effective at fooling other architectures like AlexNet and ResNet18, demonstrating the robustness of the learned filters.
- Perceptual Realism: In subjective human evaluations (Mean Opinion Score), participants could not distinguish between genuine filtered images and FilterFool’s adversarial versions 83.4% of the time.
- Inherent Robustness: Because the perturbations are structural rather than fine-grained noise, they are remarkably resistant to common defenses like JPEG compression or median blurring, which typically target high-frequency pixel fluctuations.
Conclusion
FilterFool represents a shift in adversarial research by prioritizing perceptual realism and semantic significance. By "hiding" adversarial signals within the appearance of standard image filters, the framework exposes fundamental vulnerabilities in DNNs that are not easily mitigated by standard noise-reduction defenses. Future work will focus on extending this framework to multi-label classification and exploring dynamic semantic groupings.
Summary updated