Analyzing the Noise Robustness of Deep Neural Networks

Mengchen Liu, Shixia Liu, Hang Su, Kelei Cao, Jun Zhu

This paper, presented by researchers from Tsinghua University, introduces AEVis, a visual analytics system designed to "open the black box" of Deep Neural Networks (DNNs) specifically to understand their lack of robustness against adversarial attacks.

TL;DR

AEVis is a visual analytics tool that explains why DNNs misclassify adversarial images by extracting and visualizing "datapaths"—the specific sequences of critical neurons and feature maps responsible for a prediction. By comparing the internal logic of normal vs. adversarial examples, the tool reveals exactly where and how a network’s reasoning "diverges," such as when a panda's ear is "hidden" by noise, causing a cascade of errors.

What Problem Does This Paper Address?

Deep neural networks are notoriously vulnerable to adversarial examples: images with tiny, human-imperceptible perturbations that cause the model to make confident, incorrect predictions (e.g., seeing a panda as a monkey).

While we know these attacks work, we often don't know why they work internally. Standard visualization techniques, like simple activation heatmaps, can be misleading because a neuron might be highly active for a background object that doesn't actually influence the final classification. There is a critical need for tools that can:

  1. Isolate the specific "logic chain" (datapath) the network uses for a prediction.
  2. Compare how this chain changes when noise is introduced.
  3. Scale to modern, deep architectures like ResNet-101 which have millions of neurons.

Methods

The authors propose a two-part solution: a novel extraction algorithm and a multi-level visualization interface.

1. Datapath Extraction

The authors formulate datapath extraction as a subset selection problem. They look for the smallest set of neurons that, if kept alone, would produce the same prediction as the full network.

  • The Challenge: This is an NP-complete problem.
  • The Solution: They use a "divide-and-conquer" approach, solving the problem layer-by-layer. They approximate the selection using Taylor decomposition and solve it via back-propagation, making it computationally feasible for large networks.

2. Multi-Level Visualization

To avoid visual clutter, AEVis organizes information into three hierarchies:

  • Layer Level: Uses a "Segmented DAG" (Directed Acyclic Graph) that looks like a calendar. This prevents the "long skinny strip" problem common in vertical network visualizations.
  • Feature Map Level: Uses Euler diagrams to show which feature maps are shared between normal and adversarial examples and which are unique to one.
  • Neuron Level: Employs discrepancy maps. Instead of just showing activations, these maps show what a neuron "learned" by occluding parts of the image and measuring the change in response.

Datasets

The study utilized the dataset from the NIPS 2017 Adversarial Attack Competition, which contains 1,000 high-resolution images (299x299 pixels). They tested their approach on two major architectures:

  • ResNet-101: Used for white-box attack analysis.
  • VGG-16: Used for black-box attack analysis.

Key Insights and Ideas

  • The "Diverging Point": The researchers found that adversarial noise doesn't necessarily break the network's logic immediately. Often, the internal representations of a normal and adversarial image are nearly identical for the first several layers, until they reach a "diverging point" where the logic paths split.
  • Feature Suppression vs. Creation: In the "Panda to Monkey" case study, the noise didn't just add "monkey" features; it actively suppressed the "panda ear" feature. Because the network couldn't find the ear, it failed to detect the panda's face, which then allowed a weak signal elsewhere to be misinterpreted as a monkey's face.
  • Activations vs. Criticality: High activation does not equal importance. The tool successfully filtered out neurons that were highly active (e.g., detecting a ball in a sheepdog image) but were not part of the critical datapath for the final classification.

Key Results

  • Qualitative Discovery: In a white-box attack on ResNet-101 (which had 96.3% accuracy on normal images but only 0.9% on adversarial ones), AEVis helped experts identify that the "root cause" of misclassification was the failure to detect high-level semantic parts (like ears) due to low-level noise.
  • Black-Box Insights: When analyzing a VGG-16 model, the tool showed that a panda was misclassified as a beagle because the noise in the panda's ear region perfectly mimicked the "black nose" feature the VGG-16 model used to identify dogs.
  • Expert Validation: Machine learning experts involved in the study noted that AEVis allowed them to find "subtle causes" of misclassification much faster than manual inspection of activation maps.

Limitations and Future Work

  • Scalability: While better than previous tools, visualizing networks with thousands of layers (like some modern DenseNets) still poses a screen-space challenge. The authors suggest adding "mini-maps" for navigation.
  • Cognitive Load: Comparing more than two or three datapaths simultaneously remains difficult for human users.
  • Scope: The current tool is optimized for Convolutional Neural Networks (CNNs) and image tasks. The authors suggest future work could extend this to Recurrent Neural Networks (RNNs) for text or video by "unfolding" the architecture into a DAG.

Summary updated

Abstract

Deep neural networks (DNNs) are vulnerable to maliciously generated adversarial examples. These examples are intentionally designed by making imperceptible perturbations and often mislead a DNN into making an incorrect prediction. This phenomenon means that there is significant risk in applying DNNs to safety-critical applications, such as driverless cars. To address this issue, we present a visual analytics approach to explain the primary cause of the wrong predictions introduced by adversarial examples. The key is to analyze the datapaths of the adversarial examples and compare them with those of the normal examples. A datapath is a group of critical neurons and their connections. To this end, we formulate the datapath extraction as a subset selection problem and approximately solve it based on back-propagation. A multi-level visualization consisting of a segmented DAG (layer level), an Euler diagram (feature map level), and a heat map (neuron level), has been designed to help experts investigate datapaths from the high-level layers to the detailed neuron activations. Two case studies are conducted that demonstrate the promise of our approach in support of explaining the working mechanism of adversarial examples.
BibTeX
@inproceedings{liu2018analyzing,
  author = {Mengchen Liu and Shixia Liu and Hang Su and Kelei Cao and Jun Zhu},
  title = {{Analyzing the Noise Robustness of Deep Neural Networks}},
  booktitle = {VAST},
  year = {2018},
  url = {https://openreview.net/forum?id=_Ub-xYKCD8R},
}

Loading related content...

Figure