TL;DR
This paper introduces zoNNscan, a novel index to quantify the uncertainty of a neural network's decision boundary around a given input. By measuring the entropy of model predictions within a small sampled region around an input, zoNNscan effectively highlights inputs that are close to decision boundaries, proving useful for detecting adversarial examples and corner cases.
What Problem Does This Paper Address?
Deep neural networks, while powerful, often operate with opaque decision boundaries. Understanding the geometry of these boundaries is crucial, especially as these models are deployed in critical applications like self-driving cars. Current methods lack a direct way to assess how "close" an input is to a decision boundary, which is a key characteristic of problematic inputs like adversarial examples (inputs subtly modified to cause misclassification) and corner cases (inputs that fall into ambiguous regions). This paper aims to fill this gap by providing a quantifiable measure of this boundary proximity.
Methods
The core contribution is the zoNNscan index, defined as the expected Shannon entropy of a neural network's output distribution over a small, sampled region (a "zone") around a given input.
- Shannon Entropy: A measure of uncertainty in a probability distribution. Here, it's applied to the network's confidence scores for each class.
- Zone: A small ball (defined by the L-infinity norm) around the input data point, with a specified radius
r. - Monte Carlo Sampling: Since calculating the exact expectation is computationally infeasible for high-dimensional data and complex networks, the authors propose a Monte Carlo approximation. This involves sampling
kpoints within the defined zone around the input, feeding them through the network, and averaging their output entropies. - Normalization: The entropy is normalized to output values in the range [0, 1] using a C-based logarithm (where C is the number of classes).
Datasets
The paper uses standard datasets for evaluation:
- MNIST: A dataset of handwritten digits.
- CIFAR-10: A dataset of small images across 10 object classes.
The authors also mention using models implemented in the Keras library for experiments.
Key Insights and Ideas
- Boundary Proximity as Uncertainty: The fundamental idea is that inputs near a decision boundary will lead to more uncertain (higher entropy) predictions from the neural network when sampled in their vicinity.
- Adversarial Examples are Detectable:
zoNNscanconsistently shows higher values for adversarial examples compared to their original, non-adversarial counterparts, indicating its effectiveness in flagging these problematic inputs. - Corner Cases are Also Highlighted: Inputs that cause disagreements between different neural network models (identified as corner cases) also exhibit elevated
zoNNscanvalues, suggesting its utility in identifying inputs that challenge model robustness. - Watermarking Impact: The paper demonstrates that
zoNNscancan detect changes in decision boundaries caused by techniques like model watermarking, showing that the index is sensitive to alterations in the model's internal geometry. - Radius Matters: The evolution of
zoNNscanvalues with increasing radii can reveal characteristics of surrounding regions and potentially identify problematic zones through sharp variations.
Key Results
- Adversarial Example Detection: Experiments on MLP, CNN, and IRNN models showed significantly higher
zoNNscanvalues for adversarial examples compared to standard test set inputs. Statistical tests (Kolmogorov-Smirnov) confirmed the distinguishability of these distributions with p-values < 10⁻³. - Corner Case Detection: For corner cases identified by disagreements between models,
zoNNscanvalues were also higher than for standard inputs, though with a slightly larger overlap in distributions. Statistical tests again rejected the null hypothesis. - Watermarking: After watermarking a model,
zoNNscanvalues for key inputs showed a noticeable difference compared to the unmarked model, indicating that the watermarking process altered the decision boundaries in a detectable way. - Qualitative Observations: The paper visually illustrates how
zoNNscanvalues increase as the sampling radius expands around inputs that are close to decision boundaries.
Limitations and Future Work
- Computational Cost: While Monte Carlo sampling approximates the index, it still requires running inference on multiple samples, which can be computationally intensive.
- Parameter Tuning: The choice of radius
rand the number of sampleskare parameters that need to be set appropriately for different applications and models. - Future Work: The authors suggest exploring the link between
zoNNscanvalues and other identified issues, such as new trojaning attacks on neural networks, and anticipate broader applications for the index.
Summary updated