Prototype Learning for Open Set Recognition

This direction is split into other directions: prototype-based novelty detection. Its last version stays readable and citable here.

Version 1 of 1113 papers129 conceptsfrozen 2026-10-04

Traditional machine learning operates under a "closed world" assumption: the categories the model sees during training are the only categories it will ever encounter during deployment. When a standard classifier—trained only on "cats" and "dogs"—is shown an image of a "bonsai tree," it doesn't have the vocabulary to say "I don't know." Instead, the mathematics of the Softmax function forces it to pick the "least wrong" known label with high confidence.

To solve this, researchers have moved toward a paradigm where models learn to represent classes as distinct territories in a high-dimensional feature space. By measuring the distance between a new input and these learned territories, systems can identify "unknown unknowns" and adapt to an ever-changing world.

The Failure of Softmax

In a standard neural network, the final layer typically uses a Softmax function to produce a probability distribution. Softmax is inherently competitive; for one class's probability to go up, others must go down. This works well for choosing between known options, but it fails to account for "open space risk"—the vast regions of the feature space that do not belong to any known category.

As Abhijit Bendale and Terrance E. Boult formalize in Towards Open World Recognition, an open-world system must perform two tasks simultaneously: classify known objects and detect unknown ones. If we rely solely on Softmax, the model remains "blind" to anything outside its training set.

From Probabilities to Proximity

The core intuition of this research direction is to replace relative probability with absolute distance. Instead of asking "Is this more like a cat or a dog?", the model asks "How close is this to the prototype of a cat?"

A prototype is a central representative feature vector for a category. In Kernel Null Space Methods for Novelty Detection, Paul Bodesheim and colleagues demonstrate that by projecting data into a subspace where known classes cluster tightly, we can use simple Euclidean distance to identify novelties. If a sample falls significantly far from any known class centroid, it is classified as "unknown."

This approach has been adapted across various domains:

Defining the Boundaries of Knowledge

Simply calculating distance is not enough; the model must also learn where the boundary of a class lies. This is often achieved by learning a "confidence radius" or "adaptive margin" around each prototype.

In P-ODN: Prototype-based Open Deep Network for Open Set Recognition, Yemin Shi and Yonghong Tian propose learning both a central feature vector and a specific radius for each category. This creates a "territory" for each class. If a data point falls outside all defined radii, it is rejected as unknown. Similarly, Dimity Miller and Niko Sünderhauf introduce Class Anchor Clustering, a loss function that forces known classes to cluster around fixed "anchors" in the logit space, leaving the rest of the space available for detecting unknown objects.

Carving Out Space for the Unknown

While most methods focus on pulling known samples toward a center, a more radical approach involves pushing them away from a common "unknown" area.

Guangyao Chen and Peixi Peng introduce this concept in Learning Open Set Network with Discriminative Reciprocal Points. Instead of learning what a class is, the model learns "reciprocal points"—representations of what the class is not. By pushing known classes toward the periphery of the feature space, the model creates a bounded, central "safe zone." Any sample that falls into this central region is naturally identified as unknown, providing a calibration-free way to handle open-set data.

Scaling to the Real World

The ultimate goal of prototype learning is to move AI from static benchmarks to dynamic, real-world deployments. In Large-Scale Long-Tailed Recognition in an Open World, researchers address the "natural" state of data: it is often imbalanced (long-tailed) and contains unseen categories. By using "reachability" metrics—essentially the distance to the nearest stored class centroid—models can scale their confidence. If a sample is "unreachable" from known concepts, the model's confidence is automatically dampened, preventing the overconfident errors that plague traditional closed-world systems.

By treating recognition as a geometric problem of "territories" rather than a logical problem of "labels," prototype-based methods provide a robust foundation for AI that can safely navigate the "unknown unknowns" of the real world.

Papers in this version

Showing 60 of 113 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_prototype_learning_for_open_set_recognit_910452b6,
  title = {Prototype Learning for Open Set Recognition},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/prototype-learning-for-open-set-recognition/rdr_910452b62f254171a1479735c426472d},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_prototype_learning_for_open_set_recognit_910452b6_v1,
  title = {Prototype Learning for Open Set Recognition},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/prototype-learning-for-open-set-recognition/rdr_910452b62f254171a1479735c426472d/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}