Large-Scale Open-Set Classification Protocols for ImageNet

Andres Palechor, Annesha Bhoumik, Manuel Günther

Introduction

Most modern image classifiers operate under a "closed-set" assumption, where the model expects every input to belong to one of the categories present in its training set. However, real-world applications frequently encounter "open-set" scenarios where novel, unseen classes appear. An effective Open-Set Classification (OSC) system must accurately identify known classes while simultaneously rejecting samples from categories it has never encountered.

Despite the importance of OSC, research has historically relied on small-scale, low-resolution datasets like MNIST or CIFAR-10. Furthermore, existing benchmarks often fail to distinguish between "negative" samples (classes explicitly used for rejection training) and truly "unknown" samples (classes first seen at test time). This paper addresses these gaps by introducing three rigorous, large-scale evaluation protocols based on ImageNet.

Methodology and Training Strategies

The authors evaluate three primary training strategies using a ResNet-50 architecture to establish baselines for large-scale OSC:

  1. Standard SoftMax (S): A baseline closed-set model trained only on known classes. Rejection is performed by thresholding the maximum SoftMax probability.
  2. Background Class (BG): The model includes an explicit "negative" class in the output layer. During training, the model is taught to map non-target samples into this background bucket.
  3. Entropic Open-Set (EOS): This strategy utilizes a specialized loss function that encourages the model to produce high-entropy (uniform) probability distributions for negative samples, effectively signaling uncertainty.

To better monitor performance during training, the authors propose a new validation metric, $\gamma$. This metric balances the model's accuracy on known samples with its ability to assign low confidence to negative samples, offering a more comprehensive evaluation than standard accuracy alone.

Large-Scale Evaluation Protocols

The core contribution of this work is the definition of three protocols ($P_1, P_2, P_3$) derived from the ImageNet (ILSVRC 2012) dataset. These protocols are structured using the WordNet hierarchy to test varying levels of semantic similarity between known, negative, and unknown classes.

Figure 1: Hierarchical organization of the three ImageNet protocols ($P_1, P_2, P_3$), illustrating the semantic relationships between Known, Negative, and Unknown classes.
  • Protocol $P_1$ (Easy): Known classes consist of dogs, negatives are other four-legged animals, and unknowns are non-animals (e.g., tools, fruit). This protocol features a large semantic gap.
  • Protocol $P_2$ (Medium): Known classes are a subset of hunting dogs, negatives are the remaining hunting dogs, and unknowns are other four-legged animals. The semantic gap is significantly narrowed.
  • Protocol $P_3$ (Hard): All three categories (Known, Negative, Unknown) contain a mix of animals, plants, and objects, creating high semantic overlap and a challenging classification environment.

Results and Key Insights

The experimental results reveal a stark contrast between a model's ability to reject seen negatives versus unseen unknowns. While models can learn to identify negative classes included in the training set, this capability does not always generalize to truly unknown categories, especially as semantic similarity increases.

Figure 2: Correct Classification Rate (CCR) vs. False Positive Rate (FPR) curves across the three protocols. The top row shows performance on "Negative" classes (seen during training), while the bottom row shows performance on "Unknown" classes (unseen).

The Semantic Hurdle

As shown in Figure 2, performance remains relatively high for $P_1$, where unknowns are visually distinct from knowns. However, performance significantly degrades in $P_2$ and $P_3$. This suggests that current OSC methods rely heavily on low-level visual features (like texture or fur) rather than a robust semantic understanding of "known-ness."

OOD vs. OSC

The authors argue that many tasks currently labeled as "Open-Set" are actually Out-of-Distribution (OOD) detection tasks, similar to $P_1$. True Open-Set Classification, represented by $P_3$, remains an unsolved challenge. Most existing algorithms excel at OOD detection but struggle when unknowns share semantic traits with the training data.

Strategy Comparison

The Entropic Open-Set (EOS) loss generally outperformed the Background Class (BG) approach. EOS proved more effective at rejecting negative samples without compromising closed-set accuracy. Interestingly, the study found that incorporating open-set training objectives (BG or EOS) can sometimes slightly improve performance on known classes compared to a standard SoftMax baseline.

Conclusion

This research highlights that even with advanced loss functions, current models are not yet ready for complex, real-world open-set scenarios. The reliance on specific negative samples during training creates a bottleneck; if these samples do not adequately represent the diversity of potential unknowns, the model fails to generalize. Future work must focus on developing methods that can better synthesize or represent the "unknown" space and optimizing models specifically for holistic metrics like $\gamma$.

Summary updated

Abstract

Open-Set Classification (OSC) intends to adapt closed-set classification models to real-world scenarios, where the classifier must correctly label samples of known classes while rejecting previously unseen unknown samples. Only recently, research started to investigate on algorithms that are able to handle these unknown samples correctly. Some of these approaches address OSC by including into the training set negative samples that a classifier learns to reject, expecting that these data increase the robustness of the classifier on unknown classes. Most of these approaches are evaluated on small-scale and low-resolution image datasets like MNIST, SVHN or CIFAR, which makes it difficult to assess their applicability to the real world, and to compare them among each other. We propose three open-set protocols that provide rich datasets of natural images with different levels of similarity between known and unknown classes. The protocols consist of subsets of ImageNet classes selected to provide training and testing data closer to real-world scenarios. Additionally, we propose a new validation metric that can be employed to assess whether the training of deep learning models addresses both the classification of known samples and the rejection of unknown samples. We use the protocols to compare the performance of two baseline open-set algorithms to the standard SoftMax baseline and find that the algorithms work well on negative samples that have been seen during training, and partially on out-of-distribution detection tasks, but drop performance in the presence of samples from previously unseen unknown classes.
BibTeX
@inproceedings{palechor2023large,
  author = {Andres Palechor and Annesha Bhoumik and Manuel Günther},
  title = {{Large-Scale Open-Set Classification Protocols for ImageNet}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
  doi = {10.1109/wacv56688.2023.00013},
}
Figure