PMAL: Open Set Recognition via Robust Prototype Mining

Jing Lu, Yunlu Xu, Hao Li, Zhanzhan Cheng, Yi Niu

Abstract

Open Set Recognition (OSR) presents a critical challenge for real-world machine learning: models must not only classify known categories accurately but also identify and reject unseen "unknown" classes. Traditional Prototype Learning (PL) methods often fail in this regard due to noise sensitivity and an inability to capture the diverse visual manifestations of a single class. We introduce PMAL (Prototype Mining And Learning), a novel framework that explicitly mines high-quality, diverse training samples to serve as robust class anchors. By leveraging Embedding Topology Robustness (ETR) for quality assessment and diversity-based filtering, PMAL establishes tighter class boundaries. Our evaluation shows that PMAL achieves state-of-the-art performance across multiple benchmarks, notably delivering a 16.5% AUROC improvement on the long-tailed ImageNet-LT dataset without increasing inference-time complexity.

Introduction

Standard image classifiers typically operate under a "closed set" assumption, where the model assumes every test-time object belongs to one of the categories encountered during training. In dynamic environments—such as autonomous driving or surveillance—this assumption is frequently violated. OSR requires a dual capability: maintaining high accuracy on known classes while effectively flagging unknown categories.

Motivation for PMAL: Addressing undesired learned prototypes (a) and capturing multifarious appearances (b) to better define class boundaries.

Existing Prototype Learning (PL) approaches attempt to cluster classes around central "prototypes." However, these prototypes are often learned implicitly as network parameters, leading to two significant issues:

  1. Noise Sensitivity: Prototypes can be skewed by low-quality or noisy training samples (e.g., blurry images or complex backgrounds).
  2. Lack of Diversity: A single class, such as "airplane," can have a "multifarious appearance" that a single implicit prototype cannot represent, leading to loose boundaries that misclassify unknowns as knowns.

Methodology: Prototype Mining and Learning

PMAL addresses these flaws through a two-phase framework: Prototype Mining followed by Embedding Optimization.

1. Explicit Prototype Mining

Rather than allowing the network to "invent" abstract prototype vectors, PMAL selects actual training images as anchors based on two rigorous criteria:

  • High-Quality Selection via ETR: The authors introduce Embedding Topology Robustness (ETR). This metric posits that a high-quality sample's relative position to its neighbors should remain stable across different model initializations. By training two model versions, PMAL identifies samples that maintain a consistent "neighborhood" topology, effectively filtering out noise without manual quality labels.
  • Diversity-based Filtering: To capture the full range of a class's appearance, a greedy algorithm selects a diverse set of $T$ prototypes per class. This ensures the anchors are well-distributed in the feature space, preventing redundancy.
The PMAL pipeline: High-quality candidates are mined via ETR, filtered for diversity, and then used as fixed anchors for embedding optimization.

2. Embedding Optimization

With prototypes fixed as anchors, the model is trained to map samples near them using a specialized Point-to-Set Distance metric. This metric employs a self-attention mechanism, allowing the model to dynamically "attend" to the most relevant prototypes for a given input. This grounding in real-world visual features prevents the "drifting" issues common in implicit learning and results in more stable training.

Experimental Results

PMAL was evaluated across a spectrum of benchmarks, from small-scale sets like MNIST and CIFAR to large-scale and long-tailed distributions.

  • State-of-the-Art Performance: PMAL consistently outperformed leading methods such as RPL and CPN. On TinyImageNet, PMAL-WRN achieved an Open Set AUROC of 83.1%, surpassing the previous best of 80.9%.
  • Success in Long-Tailed Scenarios: On the challenging ImageNet-LT dataset, which mimics real-world class imbalances, PMAL improved the Open Set AUROC by a massive 16.5%. This demonstrates the framework's robustness even when training data for certain classes is scarce.
  • Inference Efficiency: A key advantage of PMAL is that it adds zero additional parameters or computational overhead to the final deployed model. The complexity is handled entirely during the training phase.

Conclusion

PMAL represents a significant step forward for Open Set Recognition by shifting from implicit to explicit prototype management. By prioritizing stability and diversity in anchor selection, the framework creates a more structured and discriminative embedding space. While the training phase requires additional computation to calculate ETR scores, the resulting gains in robustness and the lack of inference-time overhead make it a highly practical solution for real-world computer vision systems.

Summary updated

Abstract

Open Set Recognition (OSR) has been an emerging topic. Besides recognizing predefined classes, the system needs to reject the unknowns. Prototype learning is a potential manner to handle the problem, as its ability to improve intra-class compactness of representations is much needed in discrimination between the known and the unknowns. In this work, we propose a novel Prototype Mining And Learning (PMAL) framework. It has a prototype mining mechanism before the phase of optimizing embedding space, explicitly considering two crucial properties, namely high-quality and diversity of the prototype set. Concretely, a set of high-quality candidates are firstly extracted from training samples based on data uncertainty learning, avoiding the interference from unexpected noise. Considering the multifarious appearance of objects even in a single category, a diversity-based strategy for prototype set filtering is proposed. Accordingly, the embedding space can be better optimized to discriminate therein the predefined classes and between known and unknowns. Extensive experiments verify the two good characteristics (i.e., high-quality and diversity) embraced in prototype mining, and show the remarkable performance of the proposed framework compared to state-of-the-arts.
BibTeX
@inproceedings{lu2022pmal,
  author = {Jing Lu and Yunlu Xu and Hao Li and Zhanzhan Cheng and Yi Niu},
  title = {{PMAL: Open Set Recognition via Robust Prototype Mining}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.1609/aaai.v36i2.20081},
}
Figure