Introduction
Clustering images into meaningful categories remains a fundamental challenge in computer vision. This task becomes significantly more difficult when dealing with fine-grained datasets, such as distinguishing between nearly identical species of birds or specific car models. Existing deep clustering methodologies typically fall into two categories, both of which struggle with fine-grained nuances:
- Discriminative Methods: These rely on heavy data augmentation (e.g., random cropping). While effective for coarse categories, these augmentations often destroy the subtle, localized features—like a specific feather pattern—required for fine-grained identification.
- Generative Methods: Approaches using VAEs or GANs attempt to reconstruct every pixel. Consequently, they often prioritize redundant background information (sky, foliage) over the subtle semantic signals of the target object.
To address these limitations, we introduce DiFiC (Diffusion Fine-grained Clustering). DiFiC leverages the latent space of pre-trained text-to-image diffusion models, which already possess a structured understanding of object semantics, to "reverse-engineer" textual proxy words that represent the essence of an image while filtering out noise.
The DiFiC Framework
The DiFiC framework is built upon three novel modules integrated with a pre-trained Stable Diffusion model. The core philosophy is to treat textual conditions as a semantic bottleneck that discards pixel-level noise while preserving identity.
1. Semantic Distillation
Instead of clustering raw pixels, DiFiC employs a Semantic Extractor to deduce a textual proxy word ($S^*$). The model is optimized to find an $S^*$ that, when placed in a prompt like "A photo of a $S^*$," allows the diffusion model to best reconstruct the original image. This process forces the model to capture the semantic "essence" of the object in a compact, word-like embedding.
2. Object Concentration
To prevent the model from being distracted by background elements, DiFiC utilizes the diffusion model’s internal cross-attention maps. These maps are used to automatically generate an object mask. By calculating reconstruction loss only on the masked foreground, the distilled $S^*$ is guaranteed to represent the object rather than the surrounding scenery.
3. Cluster Guidance
A dedicated clustering head groups these proxy words based on neighborhood similarity. To manage the high memory requirements of diffusion models, a memory bank stores historical cluster assignments. This allows for effective entropy regularization and stable training even with the small batch sizes necessitated by hardware constraints.
Technical Insight: Timestep Sampling
A key finding in this work is the importance of weighted timestep sampling. While large diffusion timesteps are useful for high-level structure, smaller timesteps (with less noise) are significantly more effective at capturing the fine-grained details essential for distinguishing similar classes.
Experimental Evaluation
DiFiC was evaluated on four challenging fine-grained benchmarks: CUB-200-2011 (birds), Stanford Cars, Oxford-IIIT Pet (dogs), and Oxford 102 Flowers. These datasets are characterized by very low inter-class variance.
Key Results
DiFiC achieved new state-of-the-art (SOTA) performance across all benchmarks, often outperforming previous methods by substantial margins:
- Stanford Cars: DiFiC reached an Accuracy (ACC) of 47.2%, a massive leap from the previous best generative method (C3-GAN at 8.3%) and discriminative method (SeCu at 10.0%).
- CUB Birds: The method achieved 31.7% ACC, surpassing the next best competitor by 9%.
- Qualitative Analysis: t-SNE visualizations confirm that DiFiC produces tighter and more distinct clusters compared to traditional contrastive learning or raw features from Stable Diffusion.
Conclusion and Future Work
DiFiC demonstrates that pre-trained generative models contain a wealth of discriminative knowledge that can be extracted through distillation into text. By using textual proxies and foreground masking, the method successfully filters out background noise and preserves fine-grained details.
While the computational overhead of using a full diffusion model is higher than traditional architectures, the paradigm of "distillation into text" opens new avenues for tasks like fine-grained classification and anomaly detection. Future work will focus on optimizing the efficiency of this framework to reduce GPU memory requirements and inference time.
Summary updated