When a deep learning model is trained under a closed-world assumption, its decision boundaries expand to fill the entire available feature space. Because the model assumes the universe consists only of its training categories, every possible input—no matter how bizarre or irrelevant—must be mapped to one of these known classes.
When deployed in the real world, however, models inevitably encounter "unknown unknowns." To handle these safely, an Open-Set Recognition (OSR) model must carve out a rejection region within its feature space. The fundamental challenge of this research direction is an inherent tension: the mechanisms used to empower a model to reject unknown inputs frequently degrade its classification precision on the known categories.
Closed-Set Framework:
[ Class A <===================> Class B ] <-- Boundaries expand to fill space
Open-Set Framework:
[ Class A <-- Rejection Region --> Class B ] <-- Carving out space compresses knowns
Understanding why this trade-off occurs requires looking closely at how open-set models alter their geometric representations, output distributions, and optimization objectives.
Why Open-Set Awareness Degrades Known Accuracy
The drop in closed-set performance during open-set adaptation is rarely a random side effect; it is typically a direct consequence of how models are forced to restructure their knowledge.
1. Feature Space Compression
To teach a network to recognize what an anomaly looks like, many frameworks explicitly train models on simulated or "proxy" unknowns. For instance, Qin Zhang et al. introduce a generative method in $G^2Pxy$ that synthesizes proxy unknown nodes using hidden-layer mixup. While this successfully teaches the network to flag out-of-distribution (OOD) data, forcing the model to accommodate these artificial classes compresses the geometric space available for the original categories, leading to a performance degradation on the known classes.
A similar geometric conflict is documented by Meghna Gummadi et al. in SHELS. By enforcing strict orthogonality and exclusivity in high-level feature representations to prevent old classes from bleeding into novel ones, the model sacrifices the representational flexibility required to maintain peak precision on its primary task.
2. The "Rejection Sink" Phenomenon
Another common strategy involves modifying the model's output layer or query vocabulary to act as a catch-all for anomalies. In the context of vision-language models, Dimity Miller et al. demonstrate that appending random or negative embeddings to a query set successfully reduces open-set errors. However, this creates a "rejection sink"—a statistical gravity well that inadvertently swallows marginal or ambiguous in-distribution samples, dragging down standard classification accuracy.
Similarly, altering the mathematical formulation of the classification scores can dilute discriminative power. As Yen-Chang Hsu et al. observe in Generalized ODIN, decomposing a network's logits into separate class-specific and domain-specific components provides robust OOD detection but causes a slight degradation in top-1 accuracy for the primary task. In dialog systems, Dmitry Lipin notes a parallel challenge, showing that optimizing the geometric alignment of intent classification models to allow for Mahalanobis-distance rejection inherently complicates the management of internal class boundaries.
3. Optimization Interference and Pseudo-Label Noise
In open-world object detection, models must simultaneously localize known objects and flag unknown foreground anomalies. To do this without explicit labels for the unknowns, frameworks often rely on self-training or automated proposal networks.
As Mısra Yavuz et al. point out in O1O, attempting to isolate unknown objects introduces low-quality pseudo-labels into the optimization loop. This training noise directly interferes with the internal representations of established categories. The model gets caught in a tug-of-war between maximizing the recall of novel objects and maintaining the precision of established categories.
Breaking the Trade-Off
Given these geometric and optimization conflicts, is the open-set trade-off an inescapable law of deep learning? Not necessarily. Recent work focuses on architectures that decouple or naturally align these competing objectives.
Bounding Classes in Probabilistic Bubbles
Instead of letting known classes expand infinitely or forcing them into tight, distorted corners to make room for a rejection class, models can bound categories within localized probabilistic regions.
Xin Sun et al. demonstrate this principle with Conditional Gaussian Distribution Learning for Open Set Recognition. By forcing different classes into distinct, bounded Gaussian "bubbles" within a Variational Autoencoder architecture, the model leaves the surrounding latent space open for anomaly rejection. Because the internal boundaries between the known classes are preserved within their respective Gaussians, the model can reject out-of-distribution samples without sacrificing closed-set accuracy.
Standard Softmax Boundaries:
[ Class A | Class B ] <-- No room for unknowns without shifting the wall
Conditional Gaussian Bubbles:
[ (Class A) . . . . . (Class B) ]
^
Rejection Space
(No boundary distortion required)
Representation Quality as a Shared Solution
An alternative perspective suggests that the trade-off can be bypassed entirely by focusing purely on representation quality rather than explicit rejection mechanisms.
Kai Han et al. challenge the need for specialized open-set architectures in Open-Set Recognition: A Good Closed-Set Classifier is All You Need. They reveal a strong positive correlation between a model's closed-set accuracy and its open-set detection capability. By optimizing standard classifiers using modern training and data-augmentation techniques, the feature representations naturally become tighter and better clustered.
Ultimately, this proves that when a model learns highly precise, compact representations of what its classes are, it gains a robust, emergent understanding of what they are not.
Key Foundational Work
Radar-based Road User Classification and Novelty Detection with Recurrent Neural Network Ensembles (2019) — contributed 2 concept elements to this direction.
Zero-Shot Detection (2018) — contributed 2 concept elements to this direction.
Large-Scale Long-Tailed Recognition in an Open World (2019) — contributed 2 concept elements to this direction.