Learning Unknowns from Unknowns: Diversified Negative Prototypes Generator for Few-shot Open-Set Recognition

Zhenyu Zhang, Guangyao Chen, Yixiong Zou, Yuhua Li, Ruixuan Li

Abstract

Few-Shot Open-Set Recognition (FSOR) requires models to classify novel "known" classes with minimal supervision while simultaneously rejecting "unknown" classes. Existing methods often struggle with a "limited imagination" problem, where negative prototypes (NPs) representing the unknown space are too specific to the few provided examples. This paper introduces the Diversified Negative Prototypes Generator (DNPG), which leverages "open weights" learned during pre-training to generate a diverse set of NPs. By incorporating Raw Class Prototypes Calibration (RPC), a Multiple NPs Generator (MNG), and a Swap Alignment (SA) module, DNPG achieves state-of-the-art performance across multiple benchmarks.

Introduction

In real-world applications, AI models frequently encounter objects outside their training distribution. FSOR addresses this by combining few-shot learning (learning from few examples) with open-set recognition (identifying unknowns). A common strategy involves using Negative Prototypes (NPs) as placeholders for the "unknown" category. However, if these NPs are derived only from the limited support set, they fail to represent the vast diversity of the unknown space. For instance, a negative prototype for a "Dalmatian" might capture dog-like features but fail to recognize a "guitar" as unknown because the guitar does not resemble the dog-specific negative prototype.

Overview of the FSOR framework across pre-training, meta-learning, and testing stages.

Methodology

The DNPG framework operates on the principle of "learning unknowns from unknowns" through several key components:

  1. Learning Open Weights: During pre-training on large base datasets, the model learns "open weights." Unlike standard weights that define a class, open weights represent the inverse—the features that exist in the space outside that class, providing a structural map of the unknown.
  2. Raw Class Prototypes Calibration (RPC): To adapt to novel classes, the model uses an attention mechanism to calibrate new support features against the established base class weights, establishing a relationship between known and novel categories.
  3. Multiple NPs Generator (MNG): Instead of a single placeholder, DNPG generates multiple, diverse NPs. This allows the model to cover various "directions" of the unknown space, from closely related but different species to completely unrelated objects. The authors found that generating five NPs provides an optimal balance.
  4. Swap Alignment (SA) Module: This module prevents the generated NPs from collapsing into a single point. By using a "Swap Task" where knowns in one task are unknowns in another, the model learns to keep NPs distinct and representative of real-world variety.
The complete DNPG architecture, illustrating the flow from backbone features to the RPC and MNG modules.

Experimental Results

DNPG was evaluated on standard benchmarks including MiniImageNet, TieredImageNet, and CIFAR-FS. It consistently outperformed state-of-the-art methods like ATT-G and GEL:

  • Accuracy and AUROC: On MiniImageNet (5-way 1-shot), DNPG achieved 69.10% accuracy and 74.18% AUROC, representing significant improvements over previous bests.
  • Robustness: The model demonstrated higher resilience to image noise (e.g., blur, artifacts), maintaining an average accuracy of 49.25% compared to 45.65% for competing models.
  • Scalability: As the complexity of the unknown space increased (up to 10 novel + 10 unknown classes), the performance gap between DNPG and other models widened, proving its superior ability to model diverse unknowns.

Discussion and Limitations

Qualitative PCA analysis confirmed that DNPG's negative prototypes are more "universal" and less biased toward the training data than previous approaches. However, the model does have limitations. It exhibits sensitivity to hyperparameters (specifically $\alpha$, $\beta$, and the number of NPs $N$), which may require tuning for new domains. Additionally, while the computational overhead is manageable, the model adds approximately 1.6M parameters over the baseline and slightly increases inference time. Future work may explore dynamic open weights that adapt during the meta-learning phase.

Conclusion

By shifting the focus from defining what a class is to understanding what it is not, DNPG provides a robust solution for Few-Shot Open-Set Recognition. The use of diversified negative prototypes and open weights allows for a more comprehensive mapping of the unknown space, leading to significant improvements in both classification and rejection tasks.

Summary updated

BibTeX
@inproceedings{zhang2024learning,
  author = {Zhenyu Zhang and Guangyao Chen and Yixiong Zou and Yuhua Li and Ruixuan Li},
  title = {{Learning Unknowns from Unknowns: Diversified Negative Prototypes Generator for Few-shot Open-Set Recognition}},
  booktitle = {ACM Multimedia},
  year = {2024},
  doi = {10.1145/3664647.3680646},
}
Figure