Deep Transformation-Invariant Clustering

Tom Monnier, Thibault Groueix, Mathieu Aubry

Introduction

Image clustering is notoriously difficult because raw pixels are highly sensitive to "nuisance" factors. For example, a picture of a "3" rotated 45 degrees looks completely different from an upright "3" in pixel space. To solve this, most modern deep learning approaches focus on representation learning: mapping images into a high-dimensional feature space where these variations are ignored. However, these features act as "black boxes," making it hard to visualize why two images ended up in the same cluster.

Older methods attempted to explicitly align images (transformation-invariant clustering), but they were computationally expensive, struggled to scale, and failed on complex deformations. Researchers from LIGM have developed an orthogonal approach: Deep Transformation-Invariant (DTI) clustering. Instead of clustering abstract features, DTI learns to "un-distort" images back to a central prototype directly in pixel space, yielding highly interpretable results that rival or beat state-of-the-art feature-based models.

The DTI Framework

Instead of just finding static cluster centers, the DTI model jointly learns a set of prototypes and a neural network that predicts how to transform a prototype to match a specific input image. This system naturally integrates into the gradient-based training of K-means and Gaussian Mixture Models (GMM) without requiring additional losses.

Classical vs DTI clustering and the deep transformation module

The core innovation is the Deep Transformation Module. For every image and potential cluster, a deep network predicts parameters for a sequence of transformations:

  1. Spatial: Affine, projective, or Thin Plate Spline (TPS) transformations handle warping and rotation.
  2. Color: Channel-wise affine transformations manage lighting and color shifts.
  3. Morphological: A novel "soft" version of dilation and erosion handles changes in stroke thickness, which is crucial for clustering digits.

To make this complex optimization work, the authors employ Curriculum Learning, starting with no transformations and gradually adding complexity (e.g., affine, then TPS, then morphological) as the model converges. They also use an empty cluster reassignment strategy to prevent the model from getting stuck in poor local minima.

Key Insights and Interpretability

Previous transformation-invariant methods tried to optimize the best transform for every single image during every step of clustering. DTI predicts them using a shared neural network. This is not only faster but acts as a form of regularization that helps the model generalize. Furthermore, DTI allows researchers to explicitly choose which invariances (color, shape, pose) the model should ignore based on the specific task.

Because the clustering happens in pixel space, the "cluster centers" are actual images (prototypes) that look like the data. You can see exactly how the model rotated, thickened, or recolored an image to make it fit a specific cluster.

Interpretable cluster prototypes and aligned samples

Results

The model was evaluated on standard benchmarks (MNIST, USPS, Fashion-MNIST, SVHN), augmented datasets (affNIST, MNIST-color), and real-world collections (Instagram images, MegaDepth, WikiPaintings).

  • State-of-the-Art Performance: On SVHN, DTI GMM achieved 63.3% accuracy, significantly outperforming previous feature-based methods like IMSAT (57.3%). On Fashion-MNIST, it reached 68.2%, beating the previous best of 66.2%.
  • Robustness to Distortion: On the "affNIST" dataset (highly rotated/scaled digits), standard deep clustering methods dropped to near-chance performance, while DTI maintained over 97% accuracy.
  • Discovery in the Wild: When applied to 15k unfiltered Instagram photos (using hashtags like #weddingkiss and #santaphoto), the model successfully discovered "iconic" poses and compositions despite massive amounts of noise and variation.

Limitations and Future Work

While the model handles 2D transformations beautifully, it struggles with strong 3D effects or complex backgrounds where a 2D prototype cannot accurately represent the scene. Additionally, the success of the model relies heavily on the curriculum learning schedule; choosing the wrong order or timing for adding transformations can lead to poor convergence.

However, the framework is highly extensible. The authors suggest that DTI isn't limited to images and could be applied to any data type—such as audio or 3D point clouds—provided the appropriate transformation modules are designed.

Summary updated

Abstract

Recent advances in image clustering typically focus on learning better deep representations. In contrast, we present an orthogonal approach that does not rely on abstract features but instead learns to predict transformations and performs clustering directly in image space. This learning process naturally fits in the gradient-based training of K-means and Gaussian mixture model, without requiring any additional loss or hyper-parameters. It leads us to two new deep transformation-invariant clustering frameworks, which jointly learn prototypes and transformations. More specifically, we use deep learning modules that enable us to resolve invariance to spatial, color and morphological transformations. Our approach is conceptually simple and comes with several advantages, including the possibility to easily adapt the desired invariance to the task and a strong interpretability of both cluster centers and assignments to clusters. We demonstrate that our novel approach yields competitive and highly promising results on standard image clustering benchmarks. Finally, we showcase its robustness and the advantages of its improved interpretability by visualizing clustering results over real photograph collections.
BibTeX
@inproceedings{monnier2020deep,
  author = {Tom Monnier and Thibault Groueix and Mathieu Aubry},
  title = {{Deep Transformation-Invariant Clustering}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=NuA41mAtevq},
}
Figure