Abstract: For any recognition system, the ability to identify novel class samples during inference is an important aspect of the system’s robustness. This problem of detecting novel class samples during inference is commonly referred to as Multiple Class Novelty Detection. In this paper, we propose a novel method that makes deep convolutional neural networks robust to novel classes. Specifically, during training one branch performs traditional classification (referred to as global inference), and the other branch provides patch-level information to keep track of the class-specific activation patterns (referred to as local inference). Both global and local branch information are combined to train a novelty detection network, which is used during inference to identify novel classes. We evaluate the proposed method on four datasets (Caltech256, CUB-200, Stanford Dogs and FounderType-200) and show that the proposed method is able to identify novel class samples better compared to the other deep convolutional neural network-based methods.
Introduction
Standard deep convolutional neural networks (DCNNs) typically operate under a "closed-set" assumption, where they are trained to categorize inputs into a fixed number of known classes. However, when deployed in the real world, these systems often encounter "novel" classes—objects the network has never seen before. In such cases, standard networks frequently misclassify these novel objects with high confidence as one of the known categories.
The challenge of Multi-Class Novelty Detection lies in the fact that novel samples are unavailable during training. While some previous methods rely on external "reference datasets" (like ImageNet) to act as a proxy for the unknown world, this is often impractical in sensitive domains like medical imaging or biometrics due to privacy and data scarcity. This research introduces a framework that achieves robustness using only the original training data.
Proposed Methodology
The core innovation is a dual-branch architecture designed to identify "spatial logic" discrepancies in classifications:
- Global Inference Network: A standard DCNN (e.g., VGG16 or AlexNet) that processes the entire image to predict a class label.
- Local Inference Network: This branch utilizes a BagNet architecture. BagNets have restricted receptive fields, allowing them to track which specific small patches of an image strongly suggest a particular category.
- Novelty Detection Network ($N$): A small Multi-Layer Perceptron (MLP) that evaluates whether the global features and the local patch-level features for the predicted class "match."
To train the system without novel images, the researchers employed a synthetic mismatch strategy. They paired global features from one image with local features of a different class from another image. This teaches the novelty detector to distinguish between correct activation patterns and the mismatched patterns typical of novel class misclassifications.
Experimental Results
The method was evaluated on four diverse benchmarks: Caltech-256 (general objects), CUB-200 (fine-grained birds), Stanford Dogs (fine-grained breeds), and FounderType-200 (Chinese font types).
The proposed method consistently outperformed existing baselines, particularly when reference data was unavailable. For instance, on the CUB-200 dataset using a VGG16 backbone, the proposed method achieved an AUC of 0.972, significantly exceeding the previous state-of-the-art (DTMND) score of 0.807 in the same setting. Qualitatively, the system correctly assigned low confidence scores to novel objects (e.g., 0.19 for a tambourine) that a baseline model misclassified with high confidence (0.76).
Key Insights and Discussion
A primary observation of this work is the Activation Discrepancy. For in-class objects, activation patterns are typically focused and logical. In contrast, when a network misclassifies a novel object, the activations are often scattered or "messy."
By learning the "intrinsic signature" of known classes through patch-level monitoring, the model becomes an expert at spotting impostors. This self-supervised approach provides significant flexibility; while it performs excellently without reference data, it can also incorporate such data via regularization to further boost performance if available.
Limitations
The main limitation remains Subtle Novelty. If a novel class is extremely similar to a known class (e.g., a specific breed of toad vs. a known frog), the activation patterns may be too similar for the discrepancy detector to trigger. Future work will focus on refining the local inference branch to capture even more granular, discriminative features to address these near-novelty cases.
Summary updated