Semantically Adversarial Learnable Filters

Ali Shahin Shamsabadi, Changjae Oh, Andrea Cavallaro

TL;DR

FilterFool is a new adversarial framework that tricks image classifiers by making images look like they’ve simply been processed with standard filters, such as gamma correction or detail enhancement. Unlike traditional attacks that add "invisible" noise, FilterFool produces large but natural-looking changes that force the AI to misclassify objects into semantically different categories (e.g., turning a "dog" into a "vehicle" rather than just another dog breed).

What Problem Does This Paper Address?

Deep Neural Networks (DNNs) are notoriously fragile. Most adversarial attacks fall into two camps: norm-bounded attacks, which add tiny, nearly invisible pixel fluctuations, and content-based attacks, which modify colors or textures more aggressively.

The authors identify two major gaps in these approaches:

  1. Perceptual Realism: Norm-bounded noise often looks like "static," while content-based attacks can produce "unrealistic" or "trippy" colors that are obvious to humans.
  2. Semantic Meaning: Most attacks consider a success if the AI predicts any wrong label. However, in a dataset like ImageNet, which has 120+ dog breeds, tricking a model into calling a "Collie" a "Shetland Sheepdog" isn't a very meaningful failure.

FilterFool aims to create "unrestricted" perturbations that look like legitimate image processing while ensuring the misclassification is semantically significant.

Methods

The core of the framework is a Fully Convolutional Neural Network (FCNN) trained to generate a perturbation ($\delta$) that, when added to an image, mimics a target filter.

The authors use a multi-task objective function with two primary components:

  • Structure Loss ($L_{Str}$): This ensures the output looks like the target filter (e.g., Log transformation, Gamma correction, or Detail Enhancement). It combines a standard $L_2$ (squared error) loss with a Structural Similarity (SSIM) index loss to preserve the "feel" and edges of the filtered image.
  • Semantic Adversarial Loss ($L_{S-Adv}$): This is the "attack" part. The authors use WordNet to group the 1000 ImageNet classes into 11 broad semantic categories (e.g., dogs, birds, structures, vehicles). The loss function specifically penalizes the model if it predicts any label within the original semantic group and rewards it for predicting a label in a different group.

By training the FCNN end-to-end, the system learns to find the specific "filter-like" adjustments that most effectively confuse the target classifier.

Datasets

  • ImageNet-1K: The primary dataset used for training and evaluation.
  • WordNet: Used to define the semantic hierarchy and group the 1000 categorical labels into 11 broader classes.
  • Validation Set: A subset of 3,000 images (3 per class) from the ImageNet validation set.

Key Insights and Ideas

  • Hiding in Plain Sight: By mimicking traditional image processing, the adversarial signal becomes "semantically transparent" to humans. We are used to seeing "gamma corrected" or "sharpened" photos, so we don't perceive the adversarial changes as malicious.
  • The Semantic Gap: The paper highlights that "categorical" success (wrong label) is easier than "semantic" success (wrong category). FilterFool is specifically designed to bridge this gap, making the model fail in a more fundamental way.
  • Inherent Robustness: Because the perturbations are essentially "filters," they are remarkably robust against common image-processing defenses like JPEG compression or median blurring. These defenses often "wash away" fine-grained noise but struggle to remove the broad, structural changes FilterFool introduces.

Key Results

  • High Effectiveness: FilterFool achieved a 100% categorical success rate on ResNet50 for several filter types (Log, Detail Enhancement).
  • Semantic Success: Even when forced to pick a semantically different category, the success rate remained extremely high (e.g., 95-100% for various filters on ResNet50).
  • Transferability: Attacks crafted for ResNet50 were surprisingly effective at fooling other models like AlexNet and ResNet18, often outperforming standard attacks like DeepFool in transferability.
  • Human Evaluation: In a subjective test (Mean Opinion Score), human observers could not distinguish between "real" filtered images and FilterFool’s adversarial versions 83.4% of the time.

Limitations and Future Work

  • Single Label Focus: The current framework is designed for single-label classification. The authors note that extending this to multi-label scenarios (where an image contains many objects) is a necessary next step.
  • Fixed Semantic Groups: The semantic groups are currently pre-defined via WordNet. Future work could involve dynamic or task-specific semantic groupings.
  • Optimization Speed: While the FCNN is fast once trained, the iterative process to find the optimal filter parameters for a specific image/model pair can be computationally intensive compared to simpler gradient-based methods.

Summary updated

Abstract

We present an adversarial framework to craft perturbations that mislead classifiers by accounting for the image content and the semantics of the labels. The proposed framework combines a structure loss and a semantic adversarial loss in a multi-task objective function to train a fully convolutional neural network. The structure loss helps generate perturbations whose type and magnitude are defined by a target image processing filter. The semantic adversarial loss considers groups of (semantic) labels to craft perturbations that prevent the filtered image from being classified with a label in the same group. We validate our framework with three different target filters, namely detail enhancement, log transformation and gamma correction filters; and evaluate the adversarially filtered images against three classifiers, ResNet50, ResNet18 and AlexNet, pre-trained on ImageNet. We show that the proposed framework generates filtered images with a high success rate, robustness, and transferability to unseen classifiers. We also discuss objective and subjective evaluations of the adversarial perturbations.
BibTeX
@article{shamsabadi2021semantically,
  author = {Ali Shahin Shamsabadi and Changjae Oh and Andrea Cavallaro},
  title = {{Semantically Adversarial Learnable Filters}},
  journal = {IEEE Trans. Image Process.},
  year = {2021},
  url = {https://openreview.net/forum?id=FEG7a5xvXYu},
}

Loading related content...

Figure