zoNNscan : a boundary-entropy index for zone inspection of neural models

Adel Jaouen, Erwan Le MerrerMCS @ IJCAI 2018 · DOI · OpenReview

TL;DR

This paper introduces zoNNscan, a novel index to quantify the uncertainty of a neural network's decision boundary around a given input. By measuring the entropy of model predictions within a small sampled region around an input, zoNNscan effectively highlights inputs that are close to decision boundaries, proving useful for detecting adversarial examples and corner cases.

What Problem Does This Paper Address?

Deep neural networks, while powerful, often operate with opaque decision boundaries. Understanding the geometry of these boundaries is crucial, especially as these models are deployed in critical applications like self-driving cars. Current methods lack a direct way to assess how "close" an input is to a decision boundary, which is a key characteristic of problematic inputs like adversarial examples (inputs subtly modified to cause misclassification) and corner cases (inputs that fall into ambiguous regions). This paper aims to fill this gap by providing a quantifiable measure of this boundary proximity.

Methods

The core contribution is the zoNNscan index, defined as the expected Shannon entropy of a neural network's output distribution over a small, sampled region (a "zone") around a given input.

  • Shannon Entropy: A measure of uncertainty in a probability distribution. Here, it's applied to the network's confidence scores for each class.
  • Zone: A small ball (defined by the L-infinity norm) around the input data point, with a specified radius r.
  • Monte Carlo Sampling: Since calculating the exact expectation is computationally infeasible for high-dimensional data and complex networks, the authors propose a Monte Carlo approximation. This involves sampling k points within the defined zone around the input, feeding them through the network, and averaging their output entropies.
  • Normalization: The entropy is normalized to output values in the range [0, 1] using a C-based logarithm (where C is the number of classes).

Datasets

The paper uses standard datasets for evaluation:

  • MNIST: A dataset of handwritten digits.
  • CIFAR-10: A dataset of small images across 10 object classes.

The authors also mention using models implemented in the Keras library for experiments.

Key Insights and Ideas

  • Boundary Proximity as Uncertainty: The fundamental idea is that inputs near a decision boundary will lead to more uncertain (higher entropy) predictions from the neural network when sampled in their vicinity.
  • Adversarial Examples are Detectable: zoNNscan consistently shows higher values for adversarial examples compared to their original, non-adversarial counterparts, indicating its effectiveness in flagging these problematic inputs.
  • Corner Cases are Also Highlighted: Inputs that cause disagreements between different neural network models (identified as corner cases) also exhibit elevated zoNNscan values, suggesting its utility in identifying inputs that challenge model robustness.
  • Watermarking Impact: The paper demonstrates that zoNNscan can detect changes in decision boundaries caused by techniques like model watermarking, showing that the index is sensitive to alterations in the model's internal geometry.
  • Radius Matters: The evolution of zoNNscan values with increasing radii can reveal characteristics of surrounding regions and potentially identify problematic zones through sharp variations.

Key Results

  • Adversarial Example Detection: Experiments on MLP, CNN, and IRNN models showed significantly higher zoNNscan values for adversarial examples compared to standard test set inputs. Statistical tests (Kolmogorov-Smirnov) confirmed the distinguishability of these distributions with p-values < 10⁻³.
  • Corner Case Detection: For corner cases identified by disagreements between models, zoNNscan values were also higher than for standard inputs, though with a slightly larger overlap in distributions. Statistical tests again rejected the null hypothesis.
  • Watermarking: After watermarking a model, zoNNscan values for key inputs showed a noticeable difference compared to the unmarked model, indicating that the watermarking process altered the decision boundaries in a detectable way.
  • Qualitative Observations: The paper visually illustrates how zoNNscan values increase as the sampling radius expands around inputs that are close to decision boundaries.

Limitations and Future Work

  • Computational Cost: While Monte Carlo sampling approximates the index, it still requires running inference on multiple samples, which can be computationally intensive.
  • Parameter Tuning: The choice of radius r and the number of samples k are parameters that need to be set appropriately for different applications and models.
  • Future Work: The authors suggest exploring the link between zoNNscan values and other identified issues, such as new trojaning attacks on neural networks, and anticipate broader applications for the index.

Summary updated

Abstract

The training of deep neural network classifiers results in decision boundaries which geometry is still not well understood. This is in direct relation with classification problems such as so called adversarial examples. We introduce zoNNscan, an index that is intended to inform on the boundary uncertainty (in terms of the presence of other classes) around one given input datapoint. It is based on confidence entropy, and is implemented through sampling in the multidimensional ball surrounding that input. We detail the zoNNscan index, give an algorithm for approximating it, and finally illustrate its benefits on four applications, including two important problems for the adoption of deep networks in critical systems: adversarial examples and corner case inputs. We highlight that zoNNscan exhibits significantly higher values than for standard inputs in those two problem classes.
BibTeX
@inproceedings{jaouen2018zonnscan,
  author = {Adel Jaouen and Erwan Le Merrer},
  title = {{zoNNscan : a boundary-entropy index for zone inspection of neural models}},
  booktitle = {MCS @ IJCAI},
  year = {2018},
  doi = {10.1007/978-3-030-89453-5\_3},
}
Figure