Large-Scale Open-Set Classification Protocols for ImageNet

Palechor Anacona, Annesha Bhoumik, Manuel Günther

Introduction

Most contemporary image classifiers operate under a "closed-set" assumption, presupposing that every image encountered during deployment belongs to one of the categories seen during training. In real-world applications, this assumption is frequently violated. A model trained to recognize specific dog breeds will inevitably encounter cats, cars, or trees.

Open-Set Classification (OSC) addresses this by enabling models to "reject" unknown samples. However, research in this area has historically relied on low-resolution, small-scale datasets like MNIST or CIFAR-10. These benchmarks often fail to capture real-world complexity because the "unknowns" are visually and semantically distinct from the "knowns." This paper introduces three new ImageNet-based protocols to provide a more rigorous framework for evaluating how deep learning models handle truly unseen data.

Methodology and Protocol Design

The authors leverage the WordNet hierarchy to organize ImageNet (ILSVRC 2012) classes into three distinct protocols ($P_1, P_2, P_3$) of increasing difficulty. These protocols are designed to control the semantic similarity between known and unknown classes.

The WordNet hierarchy used to define the three protocols (P1, P2, P3) based on semantic similarity and taxonomic distance.

The Three Protocols

  • Protocol $P_1$ (Easy): Known classes consist of all dog breeds, while unknowns are non-animal classes. This represents a high semantic distance.
  • Protocol $P_2$ (Medium): Known classes are restricted to specific hunting dogs; unknowns are other four-legged animals. These share common features like fur and anatomy.
  • Protocol $P_3$ (Hard): A diverse mix where knowns and unknowns are semantically very similar (e.g., different types of vehicles or insects), making fine-grained discrimination essential.

Crucially, the study distinguishes between Negative samples (classes seen during training as "not the target") and Unknown samples (truly unseen classes presented only during testing).

Baseline Algorithms

Three common approaches were evaluated using a ResNet-50 backbone:

  1. Standard SoftMax (S): The traditional approach, where unknowns are detected by thresholding the maximum probability score.
  2. Background Class (BG): Adds an explicit "extra" class during training that contains negative samples.
  3. Entropic Open-Set (EOS): A specialized loss function that forces negative samples toward a uniform probability distribution across all known classes, thereby increasing entropy.

To monitor training, the authors introduce the $\gamma$ validation metric. This metric balances $\gamma^+$ (average SoftMax score for correct known classes) and $\gamma^-$ (deviation from the minimum possible score for negative samples), helping researchers track if a model is learning to distinguish knowns from negatives without sacrificing accuracy.

Key Results and Analysis

The experimental results highlight a significant performance gap between identifying known negatives and rejecting truly unseen unknowns.

Performance comparison showing Correct Classification Rate (CCR) vs. False Positive Rate (FPR) across protocols for both Negative (seen) and Unknown (unseen) samples.

The "Known Unknown" Trap

Models trained with negative samples (BG and EOS) become highly proficient at rejecting those specific negative classes. However, this proficiency does not always translate to truly unseen "unknown" classes. As shown in the CCR vs. FPR curves, while EOS dominates in the easier $P_1$ protocol, its performance on unknowns in the harder $P_3$ protocol often reverts to the level of the standard SoftMax baseline.

Semantic Similarity as a Bottleneck

In $P_1$, rejecting a "non-animal" is relatively straightforward for a dog classifier. However, in $P_3$, where an unknown might be a slightly different species of bird or a different vehicle type, models struggle significantly. This suggests that OSC at scale is not merely a detection problem but a fine-grained understanding problem. While all methods maintained high closed-set accuracy (66-77%), the ability to generalize rejection to semantically similar unknowns remains the primary challenge.

Conclusion and Future Directions

The authors conclude that while their protocols highlight the limitations of current methods, existing open-set algorithms are not yet robust enough for "in-the-wild" deployment. The massive performance gap in $P_3$ between seen negatives and unseen unknowns indicates a significant generalization hurdle.

Future work should focus on negative class augmentation—finding ways to expose models to a wider variety of synthetic or sourced negatives during training to better map the boundaries of the "known" semantic space. These new ImageNet protocols provide the necessary benchmark for measuring such progress in large-scale visual recognition.

Summary updated

Abstract

Open-Set Classification (OSC) intends to adapt closed-set classification models to real-world scenarios, where the classifier must correctly label samples of known classes while rejecting previously unseen unknown samples. Only recently, research started to investigate on algorithms that are able to handle these unknown samples correctly. Some of these approaches address OSC by including into the training set negative samples that a classifier learns to reject, expecting that these data increase the robustness of the classifier on unknown classes. Most of these approaches are evaluated on small-scale and low-resolution image datasets like MNIST, SVHN or CIFAR, which makes it difficult to assess their applicability to the real world, and to compare them among each other. We propose three open-set protocols that provide rich datasets of natural images with different levels of similarity between known and unknown classes. The protocols consist of subsets of ImageNet classes selected to provide training and testing data closer to real-world scenarios. Additionally, we propose a new validation metric that can be employed to assess whether the training of deep learning models addresses both the classification of known samples and the rejection of unknown samples. We use the protocols to compare the performance of two baseline open-set algorithms to the standard SoftMax baseline and find that the algorithms work well on negative samples that have been seen during training, and partially on out-of-distribution detection tasks, but drop performance in the presence of samples from previously unseen unknown classes.
BibTeX
@inproceedings{anacona2023large,
  author = {Palechor Anacona and Annesha Bhoumik and Manuel Günther},
  title = {{Large-Scale Open-Set Classification Protocols for ImageNet}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
  doi = {10.1109/wacv56688.2023.00013},
}
Figure