Standard deep neural networks operate under a strict "closed-set" assumption: they assume that every input encountered during deployment will belong to one of the classes seen during training. When a model trained exclusively to distinguish between types of animals is shown a picture of a car, it cannot say "neither." Because the final softmax layer forces the output probabilities to sum to one, the network is compelled to confidently misclassify the novel input into an existing category.
To operate safely in real-world environments, models must be equipped with a mechanism to identify "unknown unknowns." This challenge is known as Open Set Recognition (OSR), as surveyed by Geng et al.. The core intuition behind solving OSR is to leverage the model's internal confidence metrics to establish a rejection boundary, allowing the network to abstain from prediction when an input does not fit the known training distribution.
The Baseline: Maximum Softmax Probability
The most direct way to implement an open-set rejection mechanism is to monitor the model's final prediction confidence. In a standard classifier, the final layer outputs a probability distribution across all known classes. As demonstrated by Roady et al. and Bayer et al., we can establish a threshold on the Maximum Softmax Probability (MSP). If the highest predicted class probability falls below a predefined hyperparameter threshold, the input is flagged as an out-of-distribution or unknown sample.
Standard Softmax (Closed Set):
[Class A: 0.45, Class B: 0.35, Class C: 0.20] -> Forced Choice: Class A
MSP Thresholding (Open Set, Threshold = 0.70):
Max Probability (0.45) < 0.70 -> Rejected as "Unknown"
Why does this simple approach work? When a model encounters an in-distribution sample, its features typically align closely with the cluster cores learned during training, resulting in a sharp, highly confident probability peak. Conversely, an unknown input usually falls into ambiguous regions of the feature space or far from any valid decision boundary. This causes the network's output to flatten, distributing low probabilities across multiple classes. Varshney et al. show that despite its simplicity, MSP thresholding serves as a remarkably robust baseline that frequently matches or outperforms complex architectural modifications.
Refining the Boundary: The Confidence Gap
While looking at the single highest probability peak is effective, it can fail when a model is highly conflicted between two known classes, or when an unknown sample happens to trigger a moderate response in a single category. To capture the true ambiguity of a prediction, researchers look at the relationship between the top predictions.
Instead of thresholding the maximum probability alone, a rejection logic can be implemented by calculating the difference—or margin—between the top two class probabilities. As utilized by Johansson, if the confidence gap between the first and second candidate predictions is smaller than a specific threshold, the system classifies the sample as unknown.
To ensure these margins are mathematically reliable, Bartelds et al. apply Platt scaling to transform raw classifier outputs into well-calibrated probability distributions before evaluating the top-two difference. This calibration ensures that the confidence gap reflects true statistical certainty rather than uncalibrated network logits, preventing false positives when the true class is entirely missing from the training set.
Open-Set Probability Redistribution
Post-hoc thresholding of standard softmax outputs has a fundamental limitation: deep networks are notorious for producing highly confident, incorrect predictions for "fooling" images or extreme outliers that lie completely outside the training distribution. To address this, Bendale et al. introduced OpenMax, a method that fundamentally alters how the final layer estimates probabilities.
Instead of forcing the network to distribute all its probability mass among known classes, OpenMax estimates an explicit probability for a dedicated "unknown" category. It accomplishes this by using Extreme Value Theory (EVT) to model the distribution of distances between training samples and their respective class activation vectors.
[ Input Feature ] ---> Calculate Distance to Known Class Centers
|
v
Are we too far from all centers?
/ \
[ Yes: Outlier Risk High ] [ No: Outlier Risk Low ]
| |
v v
Redistribute probability mass Retain mass in the
to the "Unknown" class slot predicted known class
When a novel input arrives, the system calculates its outlier risk based on how far it deviates from the expected training distribution. If the risk is high, the OpenMax layer actively redistributes probability mass away from the known classes and into the unknown category. This explicit open-set probability estimation, also explored by Alipour-Fanid et al. and Kim et al., prevents overconfident misclassifications by providing a formal, mathematically grounded rejection framework directly within the network's output layer.
Applications Across Domains
The necessity of probability-based thresholding spans a massive variety of real-world applications where encountering unseen data is an inevitability rather than an exception:
- Robotics and Scene Classification: In mobile robot navigation, Pronobis et al. demonstrated that confidence-based rejection allows a robot to output a null result when navigating into entirely novel rooms, preventing catastrophic localization failures. Similarly, Kwiatkowska et al. applied these thresholding strategies to acoustic scene classification to distinguish known environments from unknown ones.
- Fine-Grained Biological Identification: In tasks like fungal identification, where millions of species exist but only a fraction can be trained upon, Duan et al. used confidence thresholds to ensure that observations failing to meet a minimum certainty level are safely categorized as unknown.
- Forensics and Security: In deepfake audio detection, Wang et al. applied threshold-based decision rules to identify audio generated by unseen synthesis algorithms. In authorship attribution, Schmidt et al. configured text classifiers to abstain from making an assignment when analyzing historical texts if the output probability fell below a predefined threshold, preventing forced false attributions.
- Visual Character Checking: The authors of Towards Real-World Writing Assistance: A Chinese Character Checking Benchmark with Faked and Misspelled Characters demonstrated that monitoring confidence scores can detect entirely non-existent, handwritten "faked characters" created by stroke errors, treating low-confidence outputs as indicators of structural anomalies that do not exist in standard digital encodings.
Key Foundational Work
The Robot Vision Track at ImageCLEF 2010 (2010) — contributed 1 concept elements to this direction.
Towards Open Set Deep Networks (2016) — contributed 1 concept elements to this direction.