Latent Space Energy-based Model for Fine-grained Open Set Recognition

Wentao Bao, Qi Yu, Yu KongarXiv 2023 · DOI · arXiv

Introduction

In computer vision, traditional classification models operate under a "closed-set" assumption, where all test-time categories are known during training. However, real-world applications—ranging from autonomous driving to medical imaging—require Open Set Recognition (OSR): the ability to identify and reject previously unseen "unknown" classes. This task becomes significantly more complex in Fine-grained OSR (FineOSR), where the differences between known and unknown classes are subtle, such as distinguishing between closely related bird species.

Existing Energy-Based Models (EBMs) offer a promising framework by assigning low energy to known data and high energy to outliers. However, pixel-space EBMs often struggle to capture the intricate, low-level details necessary for fine-grained discrimination. This paper proposes a novel latent space EBM that leverages attribute-aware features and virtual outlier synthesis to define tighter decision boundaries for known classes.

Methodology

The proposed framework shifts the energy modeling from high-dimensional pixel space to a more manageable low-dimensional latent space. This transition allows the model to focus on semantic features rather than raw pixel noise while maintaining computational efficiency.

Overview of the proposed latent space EBM framework, highlighting the RAFA module, outlier synthesis, and dual EBM/Generator heads.

Key Components

  1. Residual Attribute Feature Aggregation (RAFA): To capture fine-grained details, the model employs Graph Convolutional Networks (GCNs) to disentangle global features into specific visual attributes (e.g., beak shape, wing pattern). This ensures the latent representation is sensitive to the "granularity" of the objects.
  2. Attribute-aware Information Bottleneck (AIB): This specialized loss function optimizes the latent variables to be highly expressive of both class labels and visual attributes, preventing the loss of critical discriminative information during encoding.
  3. Uncertainty-aware Virtual Outlier Synthesis (UVOS): To train the model to recognize unknowns, the authors synthesize "virtual outliers" at the boundaries of known class distributions. Unlike static methods, UVOS uses a sequential update scheme that incorporates the entire training history, leading to more stable and accurate density estimation.

Experiments and Results

The model was rigorously evaluated on benchmarks including CUB-200-2011 (birds), DeepFashion, and TinyImageNet.

Recognition Performance

On the challenging CUB dataset, the model achieved a state-of-the-art AUROC of 88.49% for detecting "Easy" unknowns when using a Vision Transformer (ViT) backbone. Crucially, it maintained a high closed-set accuracy of 93.55%, demonstrating that the ability to reject unknowns does not come at the cost of primary classification performance.

Energy score distributions on the CUB dataset. The overlap between 'Known' and 'Hard' unknown classes illustrates the difficulty of fine-grained OSR.

Generative Quality

By incorporating a generative head, the model serves as a regularizer for the classifier. The model achieved a Fréchet Inception Distance (FID) of 26.52 on CUB, a substantial improvement over the baseline SVEBM (FID 155.22). This indicates that the model learns a much richer and more realistic representation of the data manifold, which is vital for spotting subtle outliers.

Conclusion

This research demonstrates that modeling energy in a latent space, conditioned on semantic attributes, provides a robust solution for fine-grained open set recognition. By synthesizing virtual outliers and leveraging attribute-aware features, the model effectively "knows what it doesn't know." Future work may explore replacing MCMC sampling with more efficient Diffusion Models to reduce computational overhead and further enhance generative quality.

Reference: Bao, W., Yu, Q., & Kong, Y. (2023). Latent Space Energy-based Model for Fine-grained Open Set Recognition. arXiv:2309.10711v2 [cs.CV].

Summary updated

Abstract

Fine-grained open-set recognition (FineOSR) aims to recognize images belonging to classes with subtle appearance differences while rejecting images of unknown classes. A recent trend in OSR shows the benefit of generative models to discriminative unknown detection. As a type of generative model, energy-based models (EBM) are the potential for hybrid modeling of generative and discriminative tasks. However, most existing EBMs suffer from density estimation in high-dimensional space, which is critical to recognizing images from fine-grained classes. In this paper, we explore the low-dimensional latent space with energy-based prior distribution for OSR in a fine-grained visual world. Specifically, based on the latent space EBM, we propose an attribute-aware information bottleneck (AIB), a residual attribute feature aggregation (RAFA) module, and an uncertainty-based virtual outlier synthesis (UVOS) module to improve the expressivity, granularity, and density of the samples in fine-grained classes, respectively. Our method is flexible to take advantage of recent vision transformers for powerful visual classification and generation. The method is validated on both fine-grained and general visual classification datasets while preserving the capability of generating photo-realistic fake images with high resolution.
BibTeX
@misc{bao2023latent,
  author = {Wentao Bao and Qi Yu and Yu Kong},
  title = {{Latent Space Energy-based Model for Fine-grained Open Set Recognition}},
  year = {2023},
  eprint = {2309.10711},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2309.10711},
}
Figure