Abstract
Cross-modal clustering (CMC) aims to group multi-modal data by leveraging complementary information across different views. However, existing methods often struggle with the heterogeneity gap between modalities and the presence of superfluous, modality-specific information. This paper proposes Deep Mutual Information Maximin (DMIM), an end-to-end framework that simultaneously maximizes shared semantic information and filters out noise. By employing a "maximin" mutual information objective alongside an auxiliary overclustering mechanism, DMIM achieves state-of-the-art performance on several benchmark datasets.
Introduction
The goal of cross-modal clustering is to identify underlying patterns in data represented by multiple modalities, such as images paired with text descriptions. While multi-modal data provides a richer context, it introduces two significant challenges. First, the heterogeneity gap makes it difficult to align data from different mathematical spaces (e.g., pixel values vs. word embeddings). Second, each modality contains superfluous information—nuisances like background textures in images or specific vocabulary in text—that are irrelevant to the shared semantic meaning. DMIM addresses these by creating "cleaner" latent representations focused solely on shared core semantics.
Methodology
The DMIM framework consists of three primary components designed to bridge the modality gap and refine latent features.
- Multi-modal Shared Encoder: Instead of using separate encoders for each modality, DMIM utilizes a shared-parameter architecture. This forces the model to apply consistent "coding rules" across different inputs, facilitating the creation of a unified latent space.
- Variational Optimization: Direct calculation of Mutual Information (MI) is computationally difficult. DMIM uses a variational approach, employing KL divergence to provide a trainable lower bound for its objective function.
- Auxiliary Overclustering: To prevent the model from converging to degenerate solutions (e.g., grouping all data into a single cluster), an auxiliary head performs "overclustering." By predicting a much larger number of clusters than the ground truth (e.g., 10x), the model is forced to learn more granular and detailed features.
The Maximin Objective
The core of DMIM is its "maximin" objective function, which balances information retention and noise reduction.
- Maximization: The model maximizes the MI between the latent representations of the two modalities ($Z_1$ and $Z_2$). This ensures that the shared semantic "core" is captured.
- Minimization: Simultaneously, the model minimizes the MI between the original input ($X$) and its latent representation ($Z$). This acts as an information bottleneck, forcing the network to discard any information that is not shared between modalities.
Experimental Results
The researchers evaluated DMIM on four standard benchmarks: IAPR-TC12, ESP-Game, MIRFlickr, and NUS-Wide. Image features were extracted using VGG-16, and text features were obtained via BERT, both reduced to 100-dimensional vectors using PCA for computational efficiency.
DMIM consistently outperformed ten state-of-the-art baselines, including traditional methods like Canonical Correlation Analysis (CCA) and deep learning approaches like Deep Canonical Correlation Autoencoders (DCCAE). On the ESP dataset, DMIM reached an accuracy (ACC) of 0.561, a significant improvement over the 0.509 achieved by the next best deep method, DMIB. Ablation studies confirmed the importance of the maximin objective; removing the superfluous information elimination component resulted in a performance drop from 0.561 to 0.516 on the ESP dataset.
Discussion and Conclusion
The success of DMIM highlights the principle that "less is more" in representation learning. By actively forgetting modality-specific details, the model provides a cleaner signal for clustering. Furthermore, the use of overclustering as a regularizer proves effective in capturing subtle structural details that standard clustering heads might overlook. While the current work focuses on two modalities and relies on an empirical choice for the overclustering ratio, it establishes a robust foundation for future research into multi-modal information bottlenecks and scalable cross-modal clustering.
Summary updated