Open Set Recognition via Prototype Distance Thresholding

86 concepts75 papers1 proposal

The Flaw of Total Partitioning

Standard deep neural networks are remarkably confident liars when faced with the unknown. When trained on a closed set of classes, a typical network uses a Softmax layer to output a probability distribution. Geometrically, the final linear layer draws decision boundaries—hyperplanes—that partition the entire embedding space. Every single point in the space is forced to belong to some known class.

If a model trained only on "vases" and "lemons" encounters an image of a "bonsai", the hyperplanes force it into one of the existing territories. Because the activation vector must fall on one side of the line or the other, the model will confidently misclassify the novel object. This is the core vulnerability of closed-world systems: they lack a mechanism to say "none of the above."

Bounding Known Space with Prototypes

To solve this problem, researchers shift the paradigm from partitioning space to bounding space. Instead of drawing infinite lines across the feature universe, we can anchor each known class to a central point—a prototype or centroid—and draw a boundary around it.

Standard Softmax Partitioning            Prototype-Based Bounding
         |                                       _______
 Class A | Class B                              /       \
         |                                     | Class A |
 --------+--------                             \_______/
         |                                                 _______
 Class C | Class D                                        /       \
         |                                               | Class B |
                                                         \_______/
                                      [Unknown Space / Rejection Zone]

In Towards Open World Recognition, Abhijit Bendale et al. introduced this concept to manage what they termed "open space risk." Instead of relying on Softmax normalization, the system evaluates an input based on its distance to these class centroids. If an input's distance to all known class means exceeds a specific threshold, it is rejected as an outlier or classified as an unknown category.

This distance-based approach fundamentally changes the inference question. Instead of asking, "Which known class does this look most like?", the model asks, "Is this close enough to any known class to actually be one of them?"

For example, in autonomous driving, Kelvin Wong et al. in Identifying Unknown Instances for Autonomous Driving use prototypical learning to identify novel objects like stray animals or unique debris. By measuring the distance between new data points and known prototypes, the system flags uncertain points as potential novel instances, allowing a vehicle to react safely to an object even if it cannot name it.

Calibrating the Boundaries

A major challenge in prototype-based recognition is determining exactly where the boundary of a known class ends. A single, rigid threshold across all classes rarely works because some categories are naturally more diverse (forming broad, loose clusters) than others (forming tight, compact clusters).

To address this, Yemin Shi et al. developed a multi-stage thresholding mechanism in P-ODN: Prototype-based Open Deep Network for Open Set Recognition. Their network learns both a central representative feature vector and a specific confidence radius for each category, defining custom territories in the feature space.

Similarly, Hanlei Zhang et al. proposed an Adaptive Decision Boundary (ADB) framework in Deep Open Intent Classification with Adaptive Decision Boundary. ADB automatically calculates a custom radius around each known centroid during training without needing explicit examples of unknown data, ensuring tight characterization of known class boundaries.

Another elegant way to calibrate these boundaries is to account for the statistical variance of each cluster. In Open Set Synthetic Image Source Attribution, the authors calculate the distance from a query embedding to the nearest known class centroid, normalized by the standard deviation of that specific cluster. If this normalized distance exceeds a predefined threshold, the sample is safely flagged as originating from an unseen source rather than being forced into a known category.

Structuring the Latent Space

For distance-based rejection to work effectively, the underlying latent space must be explicitly structured to support it. If a network is trained with standard cross-entropy loss, it cares only about separation, not compactness. This leaves massive gaps where unknown samples can slip close to known classes.

To prevent this, modern architectures use metric learning and specialized loss functions. As surveyed by Yiming Li et al. in Open World Object Detection: A Survey, metric learning strategies constrain known classes into tight, highly localized clusters. By maximizing the distance between these clusters and potential outliers, the model leaves the unoccupied regions of the feature space open as a natural rejection zone.

For instance, Yongjuan Che et al. introduce a Multi-Relation Margin loss in Boosting Few-Shot Open-Set Recognition with Multi-Relation Margin Loss to tighten hypersphere boundaries around known classes. By carving out the surrounding feature space as a rejection zone, they achieve robust open-set recognition without needing to simulate synthetic negative data during training.

In a similar vein, Guangyao Chen et al. propose Reciprocal Point Learning in Learning Open Set Network with Discriminative Reciprocal Points. Instead of focusing solely on the center of a class, their method teaches networks to recognize known classes by learning what they are not, offering a calibration-free method for separating known and unknown data distributions.

Key Foundational Work

Familiarity Discrimination of Radar Pulses (1998) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_prototype_based_novelty_detection_4a642c7f,
  title = {prototype-based novelty detection},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/prototype-based-novelty-detection/rdr_4a642c7f1d8e4b16b5a7fe1c3b70af03},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_prototype_based_novelty_detection_4a642c7f_v1,
  title = {prototype-based novelty detection},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/prototype-based-novelty-detection/rdr_4a642c7f1d8e4b16b5a7fe1c3b70af03/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake