Abstract: Deep neural networks (DNNs) are vulnerable to maliciously generated adversarial examples. These examples are intentionally designed by making imperceptible perturbations and often mislead a DNN into making an incorrect prediction. This phenomenon means that there is significant risk in applying DNNs to safety-critical applications, such as driverless cars. To address this issue, we present a visual analytics approach to explain the primary cause of the wrong predictions introduced by adversarial examples. The key is to analyze the datapaths of the adversarial examples and compare them with those of the normal examples. A datapath is a group of critical neurons and their connections. To this end, we formulate the datapath extraction as a subset selection problem and approximately solve it based on back-propagation. A multi-level visualization consisting of a segmented DAG (layer level), an Euler diagram (feature map level), and a heat map (neuron level), has been designed to help experts investigate datapaths from the high-level layers to the detailed neuron activations. Two case studies are conducted that demonstrate the promise of our approach in support of explaining the working mechanism of adversarial examples.
Introduction
Deep neural networks (DNNs) have achieved remarkable success in various domains, yet they remain notoriously vulnerable to adversarial examples. These are inputs modified by tiny, human-imperceptible perturbations that cause a model to make confident but incorrect predictions—for instance, misidentifying a panda as a monkey. While the existence of these vulnerabilities is well-documented, the internal mechanisms that cause a network's logic to fail are often poorly understood.
Standard visualization techniques, such as simple activation heatmaps, frequently fail to distinguish between neurons that are highly active due to background noise and those that are critical to the final classification. To bridge this gap, we introduce AEVis, a visual analytics system designed to isolate and visualize the specific "logic chains" or datapaths responsible for a network's decision-making process.
Methodology
Datapath Extraction
The core of the AEVis approach is the extraction of a "datapath," defined as the minimal set of neurons and connections that drive a specific prediction. We formulate this as a subset selection problem: identifying the smallest group of neurons that, if isolated, would yield the same output as the full network.
Because this problem is NP-complete, we utilize a "divide-and-conquer" strategy. The extraction is performed layer-by-layer, approximating the selection using Taylor decomposition and solving it via back-propagation. This method allows the system to scale effectively to deep architectures like ResNet-101, which contain millions of parameters.
Multi-Level Visualization
To manage the complexity of deep architectures, AEVis organizes the extracted data into a three-tier hierarchy:
- Layer Level: A "Segmented DAG" (Directed Acyclic Graph) organizes the network structure into a calendar-like grid. This layout avoids the traditional "long skinny strip" problem, making deep networks easier to navigate.
- Feature Map Level: Euler diagrams are employed to visualize the overlap between normal and adversarial datapaths. This highlights which features are shared and which are unique to the adversarial logic.
- Neuron Level: Discrepancy maps show what a neuron has "learned" by measuring response changes when parts of the image are occluded. This provides a more accurate picture of criticality than raw activation values.
Key Insights and Results
The Diverging Point and Feature Suppression
Our analysis revealed that adversarial noise does not typically disrupt a network's logic immediately. Instead, the internal representations of normal and adversarial images often remain nearly identical through the initial layers. They eventually reach a "diverging point" where the logic paths split.
In a case study involving a panda misclassified as a monkey, we discovered that the noise did not merely add "monkey" features; it actively suppressed the "panda ear" feature. The failure to detect this high-level semantic part caused a cascade of errors, where the network eventually misinterpreted a weak signal elsewhere as a monkey's face.
Activations vs. Criticality
AEVis demonstrates that high activation does not necessarily imply importance. For example, in a sheepdog classification task, the tool successfully filtered out neurons that were highly active in response to a ball in the background, as they were not part of the critical datapath for the final "dog" classification.
Expert Validation
Evaluations using the NIPS 2017 Adversarial Attack Competition dataset on ResNet-101 and VGG-16 architectures showed that AEVis significantly aids experts. Machine learning researchers noted that the tool allowed them to identify the root causes of misclassification—such as noise mimicking a black nose in a panda's ear region—much faster than manual inspection of activation maps.
Conclusion
AEVis provides a powerful framework for "opening the black box" of DNN robustness. By focusing on datapaths rather than raw activations, it allows researchers to pinpoint exactly where and why a model's reasoning fails. Future work will focus on improving scalability for architectures with thousands of layers and extending the framework to Recurrent Neural Networks (RNNs) for text and video analysis.
Summary updated