Abstract
Open Set Recognition (OSR) presents a critical challenge for real-world machine learning: models must not only classify known categories accurately but also identify and reject unseen "unknown" classes. Traditional Prototype Learning (PL) methods often fail in this regard due to noise sensitivity and an inability to capture the diverse visual manifestations of a single class. We introduce PMAL (Prototype Mining And Learning), a novel framework that explicitly mines high-quality, diverse training samples to serve as robust class anchors. By leveraging Embedding Topology Robustness (ETR) for quality assessment and diversity-based filtering, PMAL establishes tighter class boundaries. Our evaluation shows that PMAL achieves state-of-the-art performance across multiple benchmarks, notably delivering a 16.5% AUROC improvement on the long-tailed ImageNet-LT dataset without increasing inference-time complexity.
Introduction
Standard image classifiers typically operate under a "closed set" assumption, where the model assumes every test-time object belongs to one of the categories encountered during training. In dynamic environments—such as autonomous driving or surveillance—this assumption is frequently violated. OSR requires a dual capability: maintaining high accuracy on known classes while effectively flagging unknown categories.
Existing Prototype Learning (PL) approaches attempt to cluster classes around central "prototypes." However, these prototypes are often learned implicitly as network parameters, leading to two significant issues:
- Noise Sensitivity: Prototypes can be skewed by low-quality or noisy training samples (e.g., blurry images or complex backgrounds).
- Lack of Diversity: A single class, such as "airplane," can have a "multifarious appearance" that a single implicit prototype cannot represent, leading to loose boundaries that misclassify unknowns as knowns.
Methodology: Prototype Mining and Learning
PMAL addresses these flaws through a two-phase framework: Prototype Mining followed by Embedding Optimization.
1. Explicit Prototype Mining
Rather than allowing the network to "invent" abstract prototype vectors, PMAL selects actual training images as anchors based on two rigorous criteria:
- High-Quality Selection via ETR: The authors introduce Embedding Topology Robustness (ETR). This metric posits that a high-quality sample's relative position to its neighbors should remain stable across different model initializations. By training two model versions, PMAL identifies samples that maintain a consistent "neighborhood" topology, effectively filtering out noise without manual quality labels.
- Diversity-based Filtering: To capture the full range of a class's appearance, a greedy algorithm selects a diverse set of $T$ prototypes per class. This ensures the anchors are well-distributed in the feature space, preventing redundancy.
2. Embedding Optimization
With prototypes fixed as anchors, the model is trained to map samples near them using a specialized Point-to-Set Distance metric. This metric employs a self-attention mechanism, allowing the model to dynamically "attend" to the most relevant prototypes for a given input. This grounding in real-world visual features prevents the "drifting" issues common in implicit learning and results in more stable training.
Experimental Results
PMAL was evaluated across a spectrum of benchmarks, from small-scale sets like MNIST and CIFAR to large-scale and long-tailed distributions.
- State-of-the-Art Performance: PMAL consistently outperformed leading methods such as RPL and CPN. On TinyImageNet, PMAL-WRN achieved an Open Set AUROC of 83.1%, surpassing the previous best of 80.9%.
- Success in Long-Tailed Scenarios: On the challenging ImageNet-LT dataset, which mimics real-world class imbalances, PMAL improved the Open Set AUROC by a massive 16.5%. This demonstrates the framework's robustness even when training data for certain classes is scarce.
- Inference Efficiency: A key advantage of PMAL is that it adds zero additional parameters or computational overhead to the final deployed model. The complexity is handled entirely during the training phase.
Conclusion
PMAL represents a significant step forward for Open Set Recognition by shifting from implicit to explicit prototype management. By prioritizing stability and diversity in anchor selection, the framework creates a more structured and discriminative embedding space. While the training phase requires additional computation to calculate ETR scores, the resulting gains in robustness and the lack of inference-time overhead make it a highly practical solution for real-world computer vision systems.
Summary updated