Introduction
In computer vision, traditional classification models operate under a "closed-set" assumption, where all test-time categories are known during training. However, real-world applications—ranging from autonomous driving to medical imaging—require Open Set Recognition (OSR): the ability to identify and reject previously unseen "unknown" classes. This task becomes significantly more complex in Fine-grained OSR (FineOSR), where the differences between known and unknown classes are subtle, such as distinguishing between closely related bird species.
Existing Energy-Based Models (EBMs) offer a promising framework by assigning low energy to known data and high energy to outliers. However, pixel-space EBMs often struggle to capture the intricate, low-level details necessary for fine-grained discrimination. This paper proposes a novel latent space EBM that leverages attribute-aware features and virtual outlier synthesis to define tighter decision boundaries for known classes.
Methodology
The proposed framework shifts the energy modeling from high-dimensional pixel space to a more manageable low-dimensional latent space. This transition allows the model to focus on semantic features rather than raw pixel noise while maintaining computational efficiency.
Key Components
- Residual Attribute Feature Aggregation (RAFA): To capture fine-grained details, the model employs Graph Convolutional Networks (GCNs) to disentangle global features into specific visual attributes (e.g., beak shape, wing pattern). This ensures the latent representation is sensitive to the "granularity" of the objects.
- Attribute-aware Information Bottleneck (AIB): This specialized loss function optimizes the latent variables to be highly expressive of both class labels and visual attributes, preventing the loss of critical discriminative information during encoding.
- Uncertainty-aware Virtual Outlier Synthesis (UVOS): To train the model to recognize unknowns, the authors synthesize "virtual outliers" at the boundaries of known class distributions. Unlike static methods, UVOS uses a sequential update scheme that incorporates the entire training history, leading to more stable and accurate density estimation.
Experiments and Results
The model was rigorously evaluated on benchmarks including CUB-200-2011 (birds), DeepFashion, and TinyImageNet.
Recognition Performance
On the challenging CUB dataset, the model achieved a state-of-the-art AUROC of 88.49% for detecting "Easy" unknowns when using a Vision Transformer (ViT) backbone. Crucially, it maintained a high closed-set accuracy of 93.55%, demonstrating that the ability to reject unknowns does not come at the cost of primary classification performance.
Generative Quality
By incorporating a generative head, the model serves as a regularizer for the classifier. The model achieved a Fréchet Inception Distance (FID) of 26.52 on CUB, a substantial improvement over the baseline SVEBM (FID 155.22). This indicates that the model learns a much richer and more realistic representation of the data manifold, which is vital for spotting subtle outliers.
Conclusion
This research demonstrates that modeling energy in a latent space, conditioned on semantic attributes, provides a robust solution for fine-grained open set recognition. By synthesizing virtual outliers and leveraging attribute-aware features, the model effectively "knows what it doesn't know." Future work may explore replacing MCMC sampling with more efficient Diffusion Models to reduce computational overhead and further enhance generative quality.
Reference: Bao, W., Yu, Q., & Kong, Y. (2023). Latent Space Energy-based Model for Fine-grained Open Set Recognition. arXiv:2309.10711v2 [cs.CV].
Summary updated