DiFiC: Your Diffusion Model Holds the Secret to Fine-Grained Clustering

Peng Hu (胡鹏), Ruohong Yang, Peng Hu, Xi Peng, Xiting Liu, Yunfan LiarXiv 2024 · DOI · arXiv

Introduction

Clustering images into meaningful categories remains a fundamental challenge in computer vision. This task becomes significantly more difficult when dealing with fine-grained datasets, such as distinguishing between nearly identical species of birds or specific car models. Existing deep clustering methodologies typically fall into two categories, both of which struggle with fine-grained nuances:

  1. Discriminative Methods: These rely on heavy data augmentation (e.g., random cropping). While effective for coarse categories, these augmentations often destroy the subtle, localized features—like a specific feather pattern—required for fine-grained identification.
  2. Generative Methods: Approaches using VAEs or GANs attempt to reconstruct every pixel. Consequently, they often prioritize redundant background information (sky, foliage) over the subtle semantic signals of the target object.
Conceptual comparison of clustering methods: (a) Discriminative methods lose fine-grained details, (b) Generative methods focus on redundant backgrounds, and (c) DiFiC achieves both separation and compactness by focusing on precise object semantics.

To address these limitations, we introduce DiFiC (Diffusion Fine-grained Clustering). DiFiC leverages the latent space of pre-trained text-to-image diffusion models, which already possess a structured understanding of object semantics, to "reverse-engineer" textual proxy words that represent the essence of an image while filtering out noise.

The DiFiC Framework

The DiFiC framework is built upon three novel modules integrated with a pre-trained Stable Diffusion model. The core philosophy is to treat textual conditions as a semantic bottleneck that discards pixel-level noise while preserving identity.

The DiFiC framework architecture, illustrating (a) Semantic Distillation, (b) Object Concentration via cross-attention masking, and (c) Cluster Guidance using a memory bank.

1. Semantic Distillation

Instead of clustering raw pixels, DiFiC employs a Semantic Extractor to deduce a textual proxy word ($S^*$). The model is optimized to find an $S^*$ that, when placed in a prompt like "A photo of a $S^*$," allows the diffusion model to best reconstruct the original image. This process forces the model to capture the semantic "essence" of the object in a compact, word-like embedding.

2. Object Concentration

To prevent the model from being distracted by background elements, DiFiC utilizes the diffusion model’s internal cross-attention maps. These maps are used to automatically generate an object mask. By calculating reconstruction loss only on the masked foreground, the distilled $S^*$ is guaranteed to represent the object rather than the surrounding scenery.

3. Cluster Guidance

A dedicated clustering head groups these proxy words based on neighborhood similarity. To manage the high memory requirements of diffusion models, a memory bank stores historical cluster assignments. This allows for effective entropy regularization and stable training even with the small batch sizes necessitated by hardware constraints.

Technical Insight: Timestep Sampling

A key finding in this work is the importance of weighted timestep sampling. While large diffusion timesteps are useful for high-level structure, smaller timesteps (with less noise) are significantly more effective at capturing the fine-grained details essential for distinguishing similar classes.

Experimental Evaluation

DiFiC was evaluated on four challenging fine-grained benchmarks: CUB-200-2011 (birds), Stanford Cars, Oxford-IIIT Pet (dogs), and Oxford 102 Flowers. These datasets are characterized by very low inter-class variance.

Key Results

DiFiC achieved new state-of-the-art (SOTA) performance across all benchmarks, often outperforming previous methods by substantial margins:

  • Stanford Cars: DiFiC reached an Accuracy (ACC) of 47.2%, a massive leap from the previous best generative method (C3-GAN at 8.3%) and discriminative method (SeCu at 10.0%).
  • CUB Birds: The method achieved 31.7% ACC, surpassing the next best competitor by 9%.
  • Qualitative Analysis: t-SNE visualizations confirm that DiFiC produces tighter and more distinct clusters compared to traditional contrastive learning or raw features from Stable Diffusion.

Conclusion and Future Work

DiFiC demonstrates that pre-trained generative models contain a wealth of discriminative knowledge that can be extracted through distillation into text. By using textual proxies and foreground masking, the method successfully filters out background noise and preserves fine-grained details.

While the computational overhead of using a full diffusion model is higher than traditional architectures, the paradigm of "distillation into text" opens new avenues for tasks like fine-grained classification and anomaly detection. Future work will focus on optimizing the efficiency of this framework to reduce GPU memory requirements and inference time.

Summary updated

Abstract

Fine-grained clustering is a practical yet challenging task, whose essence lies in capturing the subtle differences between instances of different classes. Such subtle differences can be easily disrupted by data augmentation or be overwhelmed by redundant information in data, leading to significant performance degradation for existing clustering methods. In this work, we introduce DiFiC a fine-grained clustering method building upon the conditional diffusion model. Distinct from existing works that focus on extracting discriminative features from images, DiFiC resorts to deducing the textual conditions used for image generation. To distill more precise and clustering-favorable object semantics, DiFiC further regularizes the diffusion target and guides the distillation process utilizing neighborhood similarity. Extensive experiments demonstrate that DiFiC outperforms both state-of-the-art discriminative and generative clustering methods on four fine-grained image clustering benchmarks. We hope the success of DiFiC will inspire future research to unlock the potential of diffusion models in tasks beyond generation. The code will be released.
BibTeX
@misc{2024dific,
  author = {Peng Hu (胡鹏) and Ruohong Yang and Peng Hu and Xi Peng and Xiting Liu and Yunfan Li},
  title = {{DiFiC: Your Diffusion Model Holds the Secret to Fine-Grained Clustering}},
  year = {2024},
  eprint = {2412.18838},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2412.18838},
}
Figure