TL;DR
This paper introduces a new type of adversarial attack that manipulates the internal representations of a deep neural network (DNN) to mimic those of a different, target image, rather than just fooling the final classification. This "feature adversary" is perceptually similar to the original image but has a representation that looks like a completely different natural image, raising fundamental questions about how DNNs truly "understand" visual data.
What Problem Does This Paper Address?
Deep neural networks (DNNs) have achieved remarkable success in image recognition. However, it's known that they can be fooled by "adversarial examples" – images with subtle, imperceptible perturbations that cause the network to misclassify them. Previous work primarily focused on generating adversarial images that trick the final classification output of the DNN. This paper addresses the question of whether these adversarial examples also manipulate the internal feature representations within the network, and if so, to what extent. The authors propose that understanding these internal representations is crucial for a deeper understanding of DNNs and their limitations.
Methods
The core of the paper is a novel optimization technique to generate "feature adversaries." Given a source image ($I_s$), a target "guide" image ($I_g$), and a trained DNN, the goal is to find a perturbed image ($I_a$) such that its internal representation at a specific layer ($k$) is very close to the representation of the guide image ($I_g$), while the perturbation to the source image ($I_s$) remains imperceptible. This is formulated as a constrained optimization problem: minimize the Euclidean distance between the representations of $I_a$ and $I_g$ at layer $k$, subject to the constraint that the $L_\infty$ norm of the difference between $I_a$ and $I_s$ is below a small threshold ($\delta$). This threshold controls the perceptual similarity. The optimization is performed using gradient-based methods (specifically, l-BFGS-b).
Datasets
The primary dataset used for experiments is ImageNet ILSVRC. The authors also mention using images from Wikipedia to increase diversity. For some experiments, they used Places205 and a hybrid dataset combining Places205 and ImageNet.
Key Insights and Ideas
- Internal Representations Matter: The most striking finding is that adversarial images can be crafted to have internal representations that closely mimic those of entirely different natural images, even if the adversarial image itself is perceptually very similar to the original.
- "Feature Adversaries" vs. "Label Adversaries": The paper distinguishes between adversarial examples that only fool the final label (label adversaries) and those that fool the internal representations (feature adversaries). Feature adversaries are presented as a more insidious and informative type of attack.
- Generic Nature of Adversarial Representations: The generated feature adversaries' representations are not outliers; they appear "generic" and indistinguishable from the representations of natural images at multiple layers of the DNN.
- Implications for DNN Understanding: This phenomenon suggests that DNNs might not be learning truly semantic or human-like representations, or that our understanding of "naturalness" in feature spaces needs re-evaluation.
- Linearity Hypothesis Questioned: The paper provides evidence against the linearity hypothesis (that adversarial examples arise from networks being "too linear") as an explanation for feature adversaries.
Key Results
- Adversarial images generated using the proposed method are perceptually very similar to the source image (with $\delta=5$ or $10$).
- Despite perceptual similarity to the source, the internal representations of these adversarial images at targeted layers are remarkably close to those of the guide images. For example, at layer FC7, the adversarial representation is often closer to the guide than to the source representation.
- In 95-100% of cases, the adversarial images are classified by the DNN as belonging to the guide image's class, not the source image's class.
- The generated feature adversaries are often nearest neighbors to their respective guide images in the feature space.
- Analysis using manifold tangent space and angular consistency measures suggests that the adversarial representations are "inliers" within the natural image manifold, meaning they share characteristics with natural image representations.
- Adversarial images generated on one network are often misclassified by other networks, but they are usually not consistently classified with the same label as the guide on different networks.
Limitations and Future Work
- Failure Cases: The optimization method sometimes fails for low-resolution images or hand-drawn characters. It also struggled with networks fine-tuned on narrow domains (e.g., Flickr Style dataset), suggesting that the nature of the representation at a given layer can be distorted by fine-tuning.
- Detecting Feature Adversaries: The paper suggests that future work could explore training models to detect these feature adversaries.
- Underlying Reasons: Further experiments are needed to fully understand the discrepancy between human and DNN representations and the precise reasons for the observed adversarial phenomena, especially concerning network depth, receptive field size, and the nature of training data.
- Linearity Hypothesis: While the paper argues against linearity for feature adversaries, further investigation into the role of linearity in DNNs and adversarial examples remains an open area.
Summary updated