Detail Reinforcement Diffusion Model: Augmentation Fine-Grained Visual Categorization in Few-Shot Conditions

Tianxu Wu, Shuo Ye, Shuhuang Chen, Qinmu Peng, Xinge You

TL;DR

This paper introduces the Detail Reinforcement Diffusion Model (DRDM) to improve fine-grained visual categorization (FGVC) in few-shot learning scenarios. DRDM tackles the challenge of data scarcity by using diffusion models for data augmentation, but crucially, it enhances this process by incorporating discriminative semantic recombination and spatial knowledge referencing to prevent feature contamination and improve model robustness.

What Problem Does This Paper Address?

Fine-grained visual categorization (FGVC) aims to distinguish between highly similar subclasses (e.g., different bird species). Traditional methods often rely on large annotated datasets and pre-trained models. However, in few-shot scenarios where only a limited number of examples are available, these methods struggle to capture the subtle differences crucial for accurate classification. While diffusion models have shown promise for data augmentation, their direct application to FGVC can lead to "feature contamination," where augmented images are either irrelevant or lack the fine details needed for discrimination. This paper addresses this by proposing a method to generate more informative and accurate augmented data for few-shot FGVC.

Methods

The proposed Detail Reinforcement Diffusion Model (DRDM) has two core components:

  • Discriminative Semantic Recombination (DSR): This module aims to improve the quality of augmented data by leveraging textual information from class labels. It reconstructs the semantic mapping between labels and images, emphasizing subclass-specific differences. This is achieved by using adapter modules within the diffusion model's architecture to transfer semantic understanding from text to image generation, guided by a Multi-Similarity loss (MS loss) and a reconstruction loss.
  • Spatial Knowledge Reference (SKR): This component addresses the issue of scattered feature distributions in high-dimensional spaces, common in few-shot learning. SKR incorporates knowledge from external, related datasets to act as a reference. By learning from diverse data distributions, it helps to define clearer and more distinct boundaries for fine-grained features, improving the model's understanding of instances and preventing overfitting. This is implemented by applying channel attention to prototype representations and query sets, and then using a classification loss combined with an SKR loss.

Datasets

The paper evaluates DRDM on three standard FGVC datasets:

  • CUB-200-2011: A dataset of bird species.
  • Stanford Dogs: A dataset of dog breeds.
  • Stanford Cars: A dataset of car models.

For the SKR component, supplementary datasets were used as external knowledge sources:

  • NABirds: For CUB.
  • Oxford-IIIT Pet: For Dogs.
  • CompCars: For Cars.

Key Insights and Ideas

  • Diffusion models are not a silver bullet for FGVC augmentation: Directly applying diffusion models can introduce irrelevant features or lose critical details, hindering few-shot FGVC performance.
  • Textual semantics are key for fine-grained augmentation: Leveraging the relationships within class labels (e.g., "Auklet" appearing in multiple bird names) can guide diffusion models to generate more semantically relevant and detailed images.
  • External knowledge is crucial for robust feature representation: Using related datasets as references helps to stabilize feature distributions and create clearer decision boundaries, especially when training data is scarce.
  • Balancing reconstruction and semantic guidance is important: The DSR module uses a weighted combination of reconstruction loss and a CLIP-based loss to ensure augmented data is both realistic and semantically aligned.

Key Results

DRDM achieved state-of-the-art (SOTA) performance on the CUB, Stanford Dogs, and Stanford Cars datasets across both 1-shot and 5-shot settings.

  • CUB: DRDM achieved 89.99% (1-shot) and 94.63% (5-shot) accuracy.
  • Stanford Dogs: DRDM achieved 72.68% (1-shot) and 80.43% (5-shot) accuracy.
  • Stanford Cars: DRDM achieved 81.53% (1-shot) and 90.03% (5-shot) accuracy.

Qualitatively, Figure 5 shows that DRDM-augmented images are more discriminative and less prone to feature contamination compared to images augmented by other diffusion-based methods. Figure 6 (t-SNE visualizations) demonstrates that DRDM leads to more compact intra-class feature distributions and better separation between classes.

Limitations and Future Work

The authors acknowledge that their method primarily relies on label similarity for feature constraints. They suggest that future work could explore additional correlated information, such as cross-hierarchical label relationships or structural information within a single instance, to further enhance generalization capabilities. They also note that while their approach improves upon existing methods, the inherent challenge of limited data in few-shot FGVC remains.

Summary updated

Abstract

The challenge in fine-grained visual categorization lies in how to explore the subtle differences between different subclasses and achieve accurate discrimination. Previous research has relied on large-scale annotated data and pre-trained deep models to achieve the objective. However, when only a limited amount of samples is available, similar methods may become less effective. Diffusion models have been widely adopted in data augmentation due to their outstanding diversity in data generation. However, the high level of detail required for fine-grained images makes it challenging for existing methods to be directly employed. To address this issue, we propose a novel approach termed the detail reinforcement diffusion model~(DRDM), which leverages the rich knowledge of large models for fine-grained data augmentation and comprises two key components including discriminative semantic recombination (DSR) and spatial knowledge reference~(SKR). Specifically, DSR is designed to extract implicit similarity relationships from the labels and reconstruct the semantic mapping between labels and instances, which enables better discrimination of subtle differences between different subclasses. Furthermore, we introduce the SKR module, which incorporates the distributions of different datasets as references in the feature space. This allows the SKR to aggregate the high-dimensional distribution of subclass features in few-shot FGVC tasks, thus expanding the decision boundary. Through these two critical components, we effectively utilize the knowledge from large models to address the issue of data scarcity, resulting in improved performance for fine-grained visual recognition tasks. Extensive experiments demonstrate the consistent performance gain offered by our DRDM.
BibTeX
@article{wu2023detail,
  author = {Tianxu Wu and Shuo Ye and Shuhuang Chen and Qinmu Peng and Xinge You},
  title = {{Detail Reinforcement Diffusion Model: Augmentation Fine-Grained Visual Categorization in Few-Shot Conditions}},
  journal = {IEEE Trans. Emerg. Top. Comput. Intell.},
  year = {2023},
  doi = {10.1109/tetci.2024.3403915},
}
Figure