End-To-End Deep Multimodal Clustering

Xianchao Zhang, Jie Mu, Linlin Zong, Xiaochun YangICME 2020 · OpenReview

Abstract

Deep multimodal clustering is challenging, since it needs to learn appropriate features for different modalities and find correct clusters by using consistency among the modalities. Existing methods treat the two problems separately, nevertheless, the optimization of one does not guarantee the optimization of the other. In this paper, we propose an end-to-end Deep Multimodal Clustering (DMMC) framework, which achieves a joint optimization of feature learning and multimodal clustering. We encourage the learned features of each modality to be cluster-oriented by autoencoder with Frobenius norm loss. Then, DMMC integrates features of different modalities by clustering integration layer and learns consensus feature by a consensus loss. The clustering result can be discovered in the consensus feature. Compared with existing clustering methods, our framework achieves superior performances measured by the standard clustering evaluation metrics.
BibTeX
@inproceedings{zhang2020end,
  author = {Xianchao Zhang and Jie Mu and Linlin Zong and Xiaochun Yang},
  title = {{End-To-End Deep Multimodal Clustering}},
  booktitle = {ICME},
  year = {2020},
  url = {https://openreview.net/forum?id=zv9DiSgBma},
}

Loading related content...

Figure