This paper, presented by researchers from Tsinghua University, introduces AEVis, a visual analytics system designed to "open the black box" of Deep Neural Networks (DNNs) specifically to understand their lack of robustness against adversarial attacks.
TL;DR
AEVis is a visual analytics tool that explains why DNNs misclassify adversarial images by extracting and visualizing "datapaths"—the specific sequences of critical neurons and feature maps responsible for a prediction. By comparing the internal logic of normal vs. adversarial examples, the tool reveals exactly where and how a network’s reasoning "diverges," such as when a panda's ear is "hidden" by noise, causing a cascade of errors.
What Problem Does This Paper Address?
Deep neural networks are notoriously vulnerable to adversarial examples: images with tiny, human-imperceptible perturbations that cause the model to make confident, incorrect predictions (e.g., seeing a panda as a monkey).
While we know these attacks work, we often don't know why they work internally. Standard visualization techniques, like simple activation heatmaps, can be misleading because a neuron might be highly active for a background object that doesn't actually influence the final classification. There is a critical need for tools that can:
- Isolate the specific "logic chain" (datapath) the network uses for a prediction.
- Compare how this chain changes when noise is introduced.
- Scale to modern, deep architectures like ResNet-101 which have millions of neurons.
Methods
The authors propose a two-part solution: a novel extraction algorithm and a multi-level visualization interface.
1. Datapath Extraction
The authors formulate datapath extraction as a subset selection problem. They look for the smallest set of neurons that, if kept alone, would produce the same prediction as the full network.
- The Challenge: This is an NP-complete problem.
- The Solution: They use a "divide-and-conquer" approach, solving the problem layer-by-layer. They approximate the selection using Taylor decomposition and solve it via back-propagation, making it computationally feasible for large networks.
2. Multi-Level Visualization
To avoid visual clutter, AEVis organizes information into three hierarchies:
- Layer Level: Uses a "Segmented DAG" (Directed Acyclic Graph) that looks like a calendar. This prevents the "long skinny strip" problem common in vertical network visualizations.
- Feature Map Level: Uses Euler diagrams to show which feature maps are shared between normal and adversarial examples and which are unique to one.
- Neuron Level: Employs discrepancy maps. Instead of just showing activations, these maps show what a neuron "learned" by occluding parts of the image and measuring the change in response.
Datasets
The study utilized the dataset from the NIPS 2017 Adversarial Attack Competition, which contains 1,000 high-resolution images (299x299 pixels). They tested their approach on two major architectures:
- ResNet-101: Used for white-box attack analysis.
- VGG-16: Used for black-box attack analysis.
Key Insights and Ideas
- The "Diverging Point": The researchers found that adversarial noise doesn't necessarily break the network's logic immediately. Often, the internal representations of a normal and adversarial image are nearly identical for the first several layers, until they reach a "diverging point" where the logic paths split.
- Feature Suppression vs. Creation: In the "Panda to Monkey" case study, the noise didn't just add "monkey" features; it actively suppressed the "panda ear" feature. Because the network couldn't find the ear, it failed to detect the panda's face, which then allowed a weak signal elsewhere to be misinterpreted as a monkey's face.
- Activations vs. Criticality: High activation does not equal importance. The tool successfully filtered out neurons that were highly active (e.g., detecting a ball in a sheepdog image) but were not part of the critical datapath for the final classification.
Key Results
- Qualitative Discovery: In a white-box attack on ResNet-101 (which had 96.3% accuracy on normal images but only 0.9% on adversarial ones), AEVis helped experts identify that the "root cause" of misclassification was the failure to detect high-level semantic parts (like ears) due to low-level noise.
- Black-Box Insights: When analyzing a VGG-16 model, the tool showed that a panda was misclassified as a beagle because the noise in the panda's ear region perfectly mimicked the "black nose" feature the VGG-16 model used to identify dogs.
- Expert Validation: Machine learning experts involved in the study noted that AEVis allowed them to find "subtle causes" of misclassification much faster than manual inspection of activation maps.
Limitations and Future Work
- Scalability: While better than previous tools, visualizing networks with thousands of layers (like some modern DenseNets) still poses a screen-space challenge. The authors suggest adding "mini-maps" for navigation.
- Cognitive Load: Comparing more than two or three datapaths simultaneously remains difficult for human users.
- Scope: The current tool is optimized for Convolutional Neural Networks (CNNs) and image tasks. The authors suggest future work could extend this to Recurrent Neural Networks (RNNs) for text or video by "unfolding" the architecture into a DAG.
Summary updated