Deep Clustering Based on a Mixture of Autoencoders

Shlomo E. Chazan, Sharon Gannot, Jacob Goldberger

Abstract

In this paper, we propose a Deep Autoencoder MIxture Clustering (DAMIC) algorithm based on a mixture of deep autoencoders where each cluster is represented by an autoencoder. A clustering network transforms the data into another space and then selects one of the clusters. Next, the autoencoder associated with this cluster is used to reconstruct the data-point. The clustering algorithm jointly learns the nonlinear data representation and the set of autoencoders. The optimal clustering is found by minimizing the reconstruction loss of the mixture of autoencoder network. Unlike other deep clustering algorithms, no regularization term is needed to avoid data collapsing to a single point. Our experimental evaluations on image and text corpora show significant improvement over state-of-the-art methods.

Introduction

Clustering high-dimensional data is a fundamental challenge in unsupervised learning because traditional distance metrics become less informative as dimensionality increases. Deep clustering addresses this by using neural networks to map data into a lower-dimensional, "clustering-friendly" embedded space.

However, standard deep clustering approaches often suffer from "clustering collapse," where the network maps all data points to a single trivial solution to minimize loss. To prevent this, existing methods rely on carefully balanced regularization terms that combine reconstruction loss with clustering loss. Tuning these hyperparameters is difficult in practice because it requires labels, which are unavailable in unsupervised tasks. This paper introduces DAMIC, an architecture that inherently resists collapse without requiring tuned regularization.

Methodology: The DAMIC Framework

DAMIC leverages a Mixture-of-Experts (MoE) framework where each cluster is represented by a full autoencoder rather than a static centroid point.

Architecture

The architecture consists of two main components:

  1. Clustering Network (The Gate): This network takes the input data and outputs a soft probability distribution (via softmax) assigning the data point to one of $k$ clusters.
  2. $k$ Autoencoders (The Experts): Each autoencoder specializes in reconstructing data belonging to its specific cluster.
DAMIC architecture for training (left) and inference (right).

Training Process

The model is trained end-to-end to minimize the expected reconstruction error across the mixture. Because the objective is to minimize error across the entire set of "experts," routing all data to a single autoencoder would result in a higher overall error by wasting the reconstructive power of the other $k-1$ networks. Thus, the architecture itself acts as a natural regularizer against collapse.

Experimental Evaluation

The authors evaluated DAMIC on a variety of datasets, including MNIST, Fashion-MNIST, and text corpora like 20 Newsgroups and RCV1-v2.

Key Results

DAMIC consistently outperformed baseline methods such as K-means, Spectral Clustering, Deep Clustering Network (DCN), and Deep Embedding Clustering (DEC):

  • MNIST: Achieved 0.87 NMI and 0.89 Accuracy, significantly beating DEC (0.80 NMI / 0.84 ACC).
  • Fashion-MNIST: Reached 0.65 NMI and 0.60 ACC, outperforming DCN (0.55 NMI / 0.50 ACC).
  • Text Data: On 20NEWS, DAMIC achieved 0.57 NMI (vs. DCN's 0.48). On RCV1, it achieved 0.62 NMI.
Visualization of DAMIC clustering results on a synthetic dataset, demonstrating superior cluster separation.

Key Insights and Discussion

  • Clusters as Data-Driven Experts: Moving from static centroids to network-based experts allows for richer, non-linear representations of cluster structures.
  • Natural Collapse Prevention: The MoE setup encourages expert specialization rather than cooperation, removing the need for hyperparameter-heavy regularization terms.
  • True Specialization: Qualitative tests showed that each autoencoder learns a specific visual "template" for its assigned cluster. For instance, on MNIST, different autoencoders reconstructed distinct, recognizable digits from dummy inputs.

While DAMIC shows state-of-the-art performance, it does introduce computational overhead as the parameter count scales linearly with the number of clusters $k$. Future work may involve extending this framework to more complex base architectures like CNNs or Transformers.

Conclusion

DAMIC provides a robust solution to the problem of clustering collapse in deep learning. By representing clusters as specialized autoencoders and optimizing for reconstruction within a mixture-of-experts framework, it achieves superior clustering performance across both image and text domains without the need for manual regularization tuning.

Summary updated

Abstract

In this paper we propose a Deep Autoencoder MIxture Clustering (DAMIC) algorithm based on a mixture of deep autoencoders where each cluster is represented by an autoencoder. A clustering network transforms the data into another space and then selects one of the clusters. Next, the autoencoder associated with this cluster is used to reconstruct the data-point. The clustering algorithm jointly learns the nonlinear data representation and the set of autoencoders. The optimal clustering is found by minimizing the reconstruction loss of the mixture of autoencoder network. Unlike other deep clustering algorithms, no regularization term is needed to avoid data collapsing to a single point. Our experimental evaluations on image and text corpora show significant improvement over state-of-the-art methods.
BibTeX
@inproceedings{chazan2019deep,
  author = {Shlomo E. Chazan and Sharon Gannot and Jacob Goldberger},
  title = {{Deep Clustering Based on a Mixture of Autoencoders}},
  booktitle = {MLSP},
  year = {2019},
  url = {https://openreview.net/forum?id=6A7cyIX-W_a},
}
Figure