Novelty Detection Through Model-Based Characterization of Neural Networks

Gukyeong Kwon, Mohit Prabhushankar, Dogancan Temel, Ghassan AlRegib

Abstract

Novelty detection is critical for the reliability of machine learning models in real-world environments. Traditional methods often rely on data-based characterizations, such as reconstruction error in autoencoders. This paper introduces a model-based characterization framework that utilizes backpropagated gradients to detect novel inputs. By measuring the "internal stress" or the magnitude of weight updates required to accommodate new data, the proposed method provides a richer, directional metric for surprise. Experimental results on MNIST, Fashion-MNIST, CIFAR-10, and CURE-TSR demonstrate that gradient-based features significantly outperform activation-based baselines in identifying both semantic class shifts and environmental distortions.

Introduction

Machine learning models often fail unpredictably when encountering data outside their training distribution. This challenge, known as novelty detection or out-of-distribution (OOD) detection, is paramount in safety-critical applications like autonomous driving or medical diagnosis. Conventional approaches typically adopt a data-based (activation) perspective, where novelty is inferred from the model's inability to reconstruct the input. However, high reconstruction error does not always correlate with novelty, and low error can mask significant distributional shifts.

This research proposes a shift toward model-based characterization. Instead of evaluating the model's output, we evaluate the model's internal state—specifically, how much the learned weights would need to change to accommodate the new input.

Methodology: From Activations to Gradients

The core innovation lies in using backpropagated gradients as a feature for novelty detection. While traditional autoencoders use reconstruction error ($\mathcal{L}$) as a scalar metric, the proposed framework extracts the gradient of this loss with respect to the model weights ($\frac{\partial \mathcal{L}}{\partial W}$).

Comparison between traditional data-based reconstruction error and the proposed model-based gradient characterization.

The Gradient Feature

When an inlier (normal data) is processed, the gradients are typically small because the model is already optimized for such patterns. Conversely, novel data generates large, directional gradients, indicating that the model's current parameters $W$ are poorly suited for the input. These high-dimensional gradient vectors provide a much richer description of abnormality than a single scalar error value.

Layer-Specific Sensitivity

The researchers observed that different layers capture different types of novelty:

  • Semantic Novelty: Gradients from layers near the latent space are most effective for detecting new object classes (e.g., a 'cat' in a model trained on 'dogs').
  • Structural Novelty: Gradients from layers near the output are better at identifying environmental distortions like rain, blur, or haze.

Experimental Evaluation

The framework was tested on four benchmarks: MNIST, Fashion-MNIST, CIFAR-10, and the CURE-TSR traffic sign dataset.

Key Results

The gradient-based approach consistently outperformed activation-based methods:

  • Accuracy: On MNIST, the gradient method achieved an AUROC of 0.953, vastly superior to reconstruction error (0.230).
  • Environmental Robustness: On CURE-TSR, the method showed significant gains in detecting distortions. For instance, in "Rain" conditions, the AUROC improved by 0.187 over the best baseline.
  • Separation: Statistical analysis confirmed that the $\ell_2$ norm of gradients provides a cleaner separation between inliers and outliers with minimal distributional overlap.

Discussion and Future Work

While highly effective, the model-based approach faces challenges. On complex datasets like CIFAR-10, the AUROC (0.582) remains lower than on simpler datasets, suggesting that visual complexity complicates gradient-based characterization. Additionally, the computational cost of performing a backpropagation pass for every inference is higher than a standard forward pass. Future research will focus on optimizing these computations for real-time deployment and developing purely unsupervised "one-class" variants of the gradient classifier.

Conclusion

By "listening to the model" through its gradients rather than just observing its outputs, we can achieve more robust and sensitive novelty detection. This model-based perspective offers a promising path toward building AI systems that are aware of their own limitations when facing the unknown.

Summary updated

Abstract

In this paper, we propose a model-based characterization of neural networks to detect novel input types and conditions. Novelty detection is crucial to identify abnormal inputs that can significantly degrade the performance of machine learning algorithms. Majority of existing studies have focused on activation-based representations to detect abnormal inputs, which limits the characterization of abnormality from a data perspective. However, a model perspective can also be informative in terms of the novelties and abnormalities. To articulate the significance of the model perspective in novelty detection, we utilize back-propagated gradients. We conduct a comprehensive analysis to compare the representation capability of gradients with that of activation and show that the gradients outperform the activation in novel class and condition detection. We validate our approach using four image recognition datasets including MNIST, Fashion-MNIST, CIFAR10, and CURE-TSR. We achieve a significant improvement on all four datasets with an average AUROC of 0.953, 0.918, 0.582, and 0.746, respectively.
BibTeX
@inproceedings{kwon2020novelty,
  author = {Gukyeong Kwon and Mohit Prabhushankar and Dogancan Temel and Ghassan AlRegib},
  title = {{Novelty Detection Through Model-Based Characterization of Neural Networks}},
  booktitle = {ICIP},
  year = {2020},
  url = {https://openreview.net/forum?id=-ZvOadKbVX},
}
Figure