Abstract
In this paper we develop a new model for deep image clustering, using convolutional neural networks and tensor kernels. The proposed Deep Tensor Kernel Clustering (DTKC) consists of a convolutional neural network (CNN), which is trained to reflect a common cluster structure at the output of its intermediate layers. Encouraging a consistent cluster structure throughout the network has the potential to guide it towards meaningful clusters, even though these clusters might appear to be nonlinear in the input space. The cluster structure is enforced through the idea of unsupervised companion objectives, where separate loss functions are attached to layers in the network. These unsupervised companion objectives are constructed based on a proposed generalization of the Cauchy-Schwarz (CS) divergence, from vectors to tensors of arbitrary rank. Generalizing the CS divergence to tensor-valued data is a crucial step, due to the tensorial nature of the intermediate representations in the CNN. Several experiments are conducted to thoroughly assess the performance of the proposed DTKC model. The results indicate that the model outperforms, or performs comparable to, a wide range of baseline algorithms. We also empirically demonstrate that our model does not suffer from objective function mismatch, which can be a problematic artifact in autoencoder-based clustering models.
Introduction: The Challenge of Objective Function Mismatch
Deep clustering models often rely on auxiliary tasks to guide the learning of feature representations. A common approach is to use an autoencoder where a reconstruction loss prevents the network from collapsing into trivial solutions. However, this introduces the Objective Function Mismatch (OFM) problem.
The conflict arises because reconstruction requires preserving fine-grained details like texture and color, whereas clustering aims to discard such noise in favor of high-level semantic features. When these objectives pull the model in different directions, clustering performance is compromised. Traditional metrics for measuring this conflict, such as Feature Drift (FD), often fail because they can be manipulated by simple loss scaling.
Methodology: Deep Tensor Kernel Clustering (DTKC)
To address the OFM problem, the authors propose Unsupervised Companion Objectives (UCOs) and structure-exploiting tensor kernels.
Unsupervised Companion Objectives (UCOs)
Instead of a decoder-based reconstruction task, UCOs are attached to the intermediate blocks of the encoder. These objectives encourage the network to form separable and compact clusters at every stage of processing. By maximizing the Cauchy-Schwarz (CS) divergence between clusters throughout the hierarchy, the model learns representations that are inherently "clustering-friendly."
Structure-Exploiting Tensor Kernels
Standard kernel methods typically require flattening multi-dimensional data into vectors, which destroys the spatial relationships inherent in CNN feature maps. DTKC utilizes tensor kernels that treat intermediate representations as tensors. By performing matricization and measuring similarity via distances on Grassmann manifolds, the model respects the multi-dimensional structure of the data, leading to more robust cluster separation.
R-OFM: A Robust Mismatch Metric
The authors introduce R-OFM, a regression-based metric that directly models the relationship between auxiliary loss and clustering performance. Unlike gradient-angle metrics, R-OFM is resistant to trivial scaling and provides a more accurate measure of how well an auxiliary task supports the final clustering goal.
Experimental Results
The DTKC model was evaluated across six diverse benchmarks, including image datasets (MNIST, Fashion-MNIST, COIL-100, USPS) and text/vector datasets (Reuters, 10 × 73k).
- Performance Gains: On the MNIST dataset, the
DDC-UCO_Tmodel achieved a significant accuracy of 96.0%, compared to 88.2% for the base DDC and 87.9% for the autoencoder-based DDC-AE. - Architectural Flexibility: The UCO approach proved effective when integrated into different architectures, such as Deep Divergence-based Clustering (DDC) and Deep Embedded Clustering (DEC).
- Reduced Mismatch: Quantitative analysis using R-OFM confirmed that UCOs maintain a much stronger positive correlation with clustering performance than traditional reconstruction losses.
Conclusion and Future Work
The proposed DTKC framework demonstrates that enforcing layer-wise consistency through UCOs and leveraging the geometric structure of data via tensor kernels can significantly improve deep clustering. While the use of tensor kernels introduces computational overhead due to Singular Value Decomposition (SVD), the gains in accuracy and the mitigation of objective function mismatch make it a compelling approach. Future research will focus on developing fully unsupervised versions of the R-OFM metric to evaluate models without relying on ground-truth labels.
Summary updated