Abstract
Recent advances in image clustering typically focus on learning better deep representations. In contrast, we present an orthogonal approach that does not rely on abstract features but instead learns to predict image transformations and performs clustering directly in image space. This learning process naturally fits in the gradient-based training of K-means and Gaussian mixture model, without requiring any additional loss or hyper-parameters. It leads us to two new deep transformation-invariant clustering frameworks, which jointly learn prototypes and transformations. More specifically, we use deep learning modules that enable us to resolve invariance to spatial, color and morphological transformations. Our approach is conceptually simple and comes with several advantages, including the possibility to easily adapt the desired invariance to the task and a strong interpretability of both cluster centers and assignments to clusters. We demonstrate that our novel approach yields competitive and highly promising results on standard image clustering benchmarks. Finally, we showcase its robustness and the advantages of its improved interpretability by visualizing clustering results over real photograph collections.
Introduction
Image clustering is a fundamental challenge in computer vision, primarily because raw pixel values are highly sensitive to "nuisance factors" such as camera angle, lighting, and object pose. Traditional methods often struggle when two images of the same object appear different due to simple rotations or color shifts.
To address this, modern research has largely shifted toward feature-based clustering, where neural networks compress images into abstract mathematical vectors. While effective, these features often act as "black boxes," making it difficult to interpret what the network has learned. This paper introduces Deep Transformation-Invariant (DTI) Clustering, a framework that returns to the intuitive idea of image alignment. By learning to mathematically "undo" transformations, DTI performs clustering directly in pixel space, resulting in both high performance and exceptional interpretability.
Methods: The DTI Framework
The core of the DTI framework is the integration of deep learning into classical clustering algorithms like K-means and Gaussian Mixture Models (GMM). Unlike previous alignment methods that were computationally expensive, DTI uses Deep Parameter Predictors—neural networks (typically a ResNet backbone) trained to predict transformation parameters directly from input images.
As shown in Figure 4, the framework utilizes a sequence of differentiable modules to handle various distortions:
- Spatial Transformations: Employs affine, projective, and Thin Plate Spline (TPS) transformations to resolve warping and pose variations.
- Color Transformations: Uses channel-wise diagonal affine transformations to handle lighting and color shifts.
- Morphological Transformations: Introduces a novel "soft" version of dilation and erosion to handle changes in stroke thickness, which is particularly useful for handwritten digits.
The cluster prototypes (the "average" image for each group) and the transformation networks are optimized jointly. To ensure stability, the authors employ curriculum learning, starting with simple transformations and gradually introducing more complex ones.
Key Results and Interpretability
DTI achieves state-of-the-art results on several benchmarks, including Fashion-MNIST (68.2% accuracy) and SVHN (63.3% accuracy). A major strength of the model is its robustness; on the heavily distorted "affNIST" dataset, DTI maintained 97% accuracy, whereas popular feature-based methods like IIC dropped to 57.6%.
Beyond numerical performance, DTI offers superior interpretability. Because the model operates in pixel space, the learned cluster centers are sharp, recognizable images rather than abstract vectors.
As illustrated in Figure 6, DTI produces significantly clearer prototypes than standard GMMs. This allows researchers to visually verify what each cluster represents. Furthermore, the model's explicit control over transformations allows users to tailor invariances to specific tasks. When applied to unfiltered Instagram data (e.g., #weddingkiss), the model successfully identified iconic poses and visual trends without any human labels.
Conclusion
Deep Transformation-Invariant Clustering demonstrates that pixel-space clustering is not only viable but can outperform complex feature-based embeddings when paired with a strong alignment mechanism. By jointly learning transformations and prototypes, DTI provides a scalable, efficient, and highly interpretable solution for modern image clustering. Future work may involve extending these transformation modules to handle complex 3D viewpoint changes and applying the framework to non-image data types.
Summary updated