Introduction
Most modern image classifiers operate under a "closed-set" assumption, where the model expects every input to belong to one of the categories present in its training set. However, real-world applications frequently encounter "open-set" scenarios where novel, unseen classes appear. An effective Open-Set Classification (OSC) system must accurately identify known classes while simultaneously rejecting samples from categories it has never encountered.
Despite the importance of OSC, research has historically relied on small-scale, low-resolution datasets like MNIST or CIFAR-10. Furthermore, existing benchmarks often fail to distinguish between "negative" samples (classes explicitly used for rejection training) and truly "unknown" samples (classes first seen at test time). This paper addresses these gaps by introducing three rigorous, large-scale evaluation protocols based on ImageNet.
Methodology and Training Strategies
The authors evaluate three primary training strategies using a ResNet-50 architecture to establish baselines for large-scale OSC:
- Standard SoftMax (S): A baseline closed-set model trained only on known classes. Rejection is performed by thresholding the maximum SoftMax probability.
- Background Class (BG): The model includes an explicit "negative" class in the output layer. During training, the model is taught to map non-target samples into this background bucket.
- Entropic Open-Set (EOS): This strategy utilizes a specialized loss function that encourages the model to produce high-entropy (uniform) probability distributions for negative samples, effectively signaling uncertainty.
To better monitor performance during training, the authors propose a new validation metric, $\gamma$. This metric balances the model's accuracy on known samples with its ability to assign low confidence to negative samples, offering a more comprehensive evaluation than standard accuracy alone.
Large-Scale Evaluation Protocols
The core contribution of this work is the definition of three protocols ($P_1, P_2, P_3$) derived from the ImageNet (ILSVRC 2012) dataset. These protocols are structured using the WordNet hierarchy to test varying levels of semantic similarity between known, negative, and unknown classes.
- Protocol $P_1$ (Easy): Known classes consist of dogs, negatives are other four-legged animals, and unknowns are non-animals (e.g., tools, fruit). This protocol features a large semantic gap.
- Protocol $P_2$ (Medium): Known classes are a subset of hunting dogs, negatives are the remaining hunting dogs, and unknowns are other four-legged animals. The semantic gap is significantly narrowed.
- Protocol $P_3$ (Hard): All three categories (Known, Negative, Unknown) contain a mix of animals, plants, and objects, creating high semantic overlap and a challenging classification environment.
Results and Key Insights
The experimental results reveal a stark contrast between a model's ability to reject seen negatives versus unseen unknowns. While models can learn to identify negative classes included in the training set, this capability does not always generalize to truly unknown categories, especially as semantic similarity increases.
The Semantic Hurdle
As shown in Figure 2, performance remains relatively high for $P_1$, where unknowns are visually distinct from knowns. However, performance significantly degrades in $P_2$ and $P_3$. This suggests that current OSC methods rely heavily on low-level visual features (like texture or fur) rather than a robust semantic understanding of "known-ness."
OOD vs. OSC
The authors argue that many tasks currently labeled as "Open-Set" are actually Out-of-Distribution (OOD) detection tasks, similar to $P_1$. True Open-Set Classification, represented by $P_3$, remains an unsolved challenge. Most existing algorithms excel at OOD detection but struggle when unknowns share semantic traits with the training data.
Strategy Comparison
The Entropic Open-Set (EOS) loss generally outperformed the Background Class (BG) approach. EOS proved more effective at rejecting negative samples without compromising closed-set accuracy. Interestingly, the study found that incorporating open-set training objectives (BG or EOS) can sometimes slightly improve performance on known classes compared to a standard SoftMax baseline.
Conclusion
This research highlights that even with advanced loss functions, current models are not yet ready for complex, real-world open-set scenarios. The reliance on specific negative samples during training creates a bottleneck; if these samples do not adequately represent the diversity of potential unknowns, the model fails to generalize. Future work must focus on developing methods that can better synthesize or represent the "unknown" space and optimizing models specifically for holistic metrics like $\gamma$.
Summary updated