Abstract
Multi-modal clustering (MMC) aims to group data objects by identifying consistent patterns across multiple formats, such as images and text. However, existing methods often struggle with modality-specific redundancy and insufficient guidance between feature learning and cluster assignment. This paper presents the Multi-aspect Self-guided Deep Information Bottleneck (MSDIB) framework. Grounded in Information Bottleneck (IB) theory, MSDIB optimizes for a representation that is as compact as possible while remaining highly informative about the target cluster structure. Through a dual-module approach—Information Compression and Information Preservation—and a multi-level self-guidance mechanism, the framework effectively filters noise and captures shared signals. Experimental results on five benchmark datasets demonstrate that MSDIB consistently achieves state-of-the-art performance and exhibits high robustness to hyperparameter variations.
Introduction
In the era of big data, information is rarely confined to a single modality. A typical social media entry, for instance, comprises an image, a textual caption, and various tags. Multi-modal clustering seeks to leverage these diverse views to uncover underlying structures. Two primary challenges hinder current MMC approaches: redundancy and weak guidance. Modalities often contain "private" noise that is irrelevant to the global clustering task. Furthermore, many models fail to utilize the global clustering results to refine individual modality representations, missing the opportunity for the global structure to "guide" local feature learning.
The MSDIB Framework
The proposed MSDIB framework utilizes the Information Bottleneck (IB) principle to find a representation that balances compression and preservation. The goal is to create a latent space that is "cluster-friendly" by stripping away irrelevant details while maintaining the core signals shared across modalities.
As illustrated in the architecture, the framework is divided into two core components:
- Information Compression (IC): This module employs deep encoders to extract compact representations. It implements mutual compression by minimizing mutual information between modalities, thereby stripping away redundant, modality-specific data that does not contribute to the shared cluster structure.
- Information Preservation (IP): This module ensures the retention of critical shared signals by maximizing mutual information between private features and a fused global representation.
A key innovation of MSDIB is its multi-level self-guidance mechanism. This includes mutual guidance between modalities, guidance from fused global features to private features, and guidance from global cluster assignments (pseudo-labels) to local assignments. This creates a robust feedback loop that ensures all parts of the model converge toward a cohesive clustering result.
Experimental Evaluation
The model was evaluated on five widely recognized multi-modal datasets: Caltech-2V, ESP-Game, IAPR, MIRFlickr, and NUS-Wide. These datasets provide a range of image-text pairs and multi-feature image data, offering a comprehensive test of the framework's versatility across different data types and scales.
Key Results and Discussion
MSDIB consistently outperformed state-of-the-art (SOTA) methods, such as MFLVC and DIVICE, across all five datasets in terms of Accuracy (ACC) and Normalized Mutual Information (NMI). A notable highlight was the ESP-Game dataset, where MSDIB achieved a 14.1% improvement in Accuracy over the next best-performing method.
Beyond raw performance, the framework demonstrated significant stability. Parameter sensitivity analysis indicates that MSDIB is relatively insensitive to its trade-off hyperparameters ($\alpha$ and $\beta$). As shown in the 3D visualizations above, the model maintains high accuracy across a wide range of settings, reducing the need for exhaustive hyperparameter tuning in practical applications.
Conclusion
MSDIB offers a theoretically grounded and empirically robust solution for multi-modal clustering. By integrating deep learning with Information Bottleneck theory, it successfully balances noise reduction with feature preservation. While the current framework is designed for aligned data, future work will focus on extending MSDIB to handle incomplete or unaligned multi-modal datasets, which are common in messy, real-world environments.
Summary updated