Detecting Adversarial Examples and Other Misclassifications in Neural Networks by Introspection

Marcin Detyniecki, Ph.D., Jonathan Aigrain, Marcin DetynieckiarXiv 2019 · arXiv

Introduction

Modern deep learning models suffer from a persistent "confidently wrong" problem. Despite their high accuracy on standard benchmarks, neural networks are notoriously overconfident, often assigning high probabilities to incorrect predictions, out-of-distribution (OOD) data, or nonsensical noise. This paper introduces Introspection-Net, a secondary network designed to monitor the primary model's internal state to accurately predict classification correctness.

The Problem: The Softmax Information Bottleneck

The standard softmax layer, which converts raw network outputs into a probability distribution, acts as an information bottleneck. By normalizing outputs, it exaggerates small relative differences between classes while discarding the absolute magnitude of the network's internal activations. This leads to high-confidence errors in three main scenarios:

  1. Regular Errors: Misclassifications of standard, in-distribution data.
  2. Out-of-Distribution (OOD) Data: Inputs the model has never seen (e.g., random noise or images from a different domain).
  3. Adversarial Examples: Images engineered with subtle noise specifically to trick a model into a high-confidence wrong answer.

Methodology: Introspection-Net

The authors propose a technique called introspection. Instead of relying on the final probability score, they train a small auxiliary model called Introspection-Net to look at the raw "logits" (the values produced by the last layer before the softmax function) of a pre-trained, frozen classifier.

Architecture and Training

The Introspection-Net is a simple 3-layer dense neural network (128 neurons per layer) utilizing ReLU activations, Dropout, and Batch Normalization. Its task is to perform a regression that predicts a "confidence value" between 0 (incorrect) and 1 (correct). To ensure robustness, the network is trained on a balanced dataset containing correctly classified images and various "misclassifications" generated via data augmentation (black patches), OOD datasets, and adversarial attack methods like the Fast Gradient Sign Method (FGSM) and the Basic Iterative Method (BIM).

Why Logit Magnitude Matters

The core insight of the paper is that correctly classified images consistently produce higher-magnitude activation patterns than errors or adversarial examples. While adversarial attacks are highly effective at "fooling" the final softmax output by manipulating relative differences, they often fail to mimic the high-magnitude logit patterns characteristic of genuine, clean data.

Comparison of Softmax vs. Logit distributions. While softmax values (right) for various inputs are often indistinguishable near 1.0, raw logit values (left) provide clear separation between clean data (MNIST) and adversarial or out-of-distribution samples.

As shown in the scatter plots above, softmax values for clean data (MNIST) and errors are often clustered near 1.0, making them indistinguishable. However, looking at the raw logit values reveals a distinct separation, allowing the Introspection-Net to catch discrepancies that the primary model misses.

Experimental Results

The researchers evaluated their approach using two primary setups: a custom CNN for MNIST and a Wide Residual Network (WR-28-8) for CIFAR-10. Introspection-Net was compared against the standard "Softmax Baseline" and the "Trust Score" (a popular 1-NN based metric).

Performance comparison across OOD and adversarial datasets. The proposed Introspection-Net significantly outperforms the Softmax Baseline and Trust Score across nearly all metrics, particularly in detecting noise and iterative attacks.

Key Findings

  • OOD Detection: In MNIST experiments, Introspection-Net achieved a 0.0% False Positive Rate (FPR) at a 95% True Positive Rate for Gaussian noise. In contrast, the Softmax Baseline had a 100% FPR, meaning it could not distinguish noise from digits at all.
  • Adversarial Robustness: For the BIM attack on CIFAR-10, Introspection-Net reached an AUROC of 97.9%, compared to a dismal 14.4% for the Softmax Baseline.
  • Generalization: The method proved effective across different architectures, suggesting that logit distribution patterns are a fundamental property of how these networks learn.

Limitations and Future Work

Despite its success, Introspection-Net has limitations. It currently requires adversarial training on the specific types of errors it is meant to detect; if a new, unknown attack is used, the network might fail to recognize the pattern. Additionally, unlike model-agnostic methods like Trust Score, Introspection-Net is tied to the specific logits of a single pre-trained model. Future research will explore whether introspection can be achieved using only in-distribution data and how these principles apply to other domains like Natural Language Processing.

Summary updated

BibTeX
@misc{phd2019detecting,
  author = {Marcin Detyniecki, Ph.D. and Jonathan Aigrain and Marcin Detyniecki},
  title = {{Detecting Adversarial Examples and Other Misclassifications in Neural Networks by Introspection}},
  year = {2019},
  eprint = {1905.09186},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1905.09186},
}
Figure