Multi-aspect Self-guided Deep Information Bottleneck for Multi-modal Clustering

Shizhe Hu, Jiahao Fan, Guoliang Zou, Yangdong Ye

Abstract

Multi-modal clustering (MMC) aims to group data objects by identifying consistent patterns across multiple formats, such as images and text. However, existing methods often struggle with modality-specific redundancy and insufficient guidance between feature learning and cluster assignment. This paper presents the Multi-aspect Self-guided Deep Information Bottleneck (MSDIB) framework. Grounded in Information Bottleneck (IB) theory, MSDIB optimizes for a representation that is as compact as possible while remaining highly informative about the target cluster structure. Through a dual-module approach—Information Compression and Information Preservation—and a multi-level self-guidance mechanism, the framework effectively filters noise and captures shared signals. Experimental results on five benchmark datasets demonstrate that MSDIB consistently achieves state-of-the-art performance and exhibits high robustness to hyperparameter variations.

Introduction

In the era of big data, information is rarely confined to a single modality. A typical social media entry, for instance, comprises an image, a textual caption, and various tags. Multi-modal clustering seeks to leverage these diverse views to uncover underlying structures. Two primary challenges hinder current MMC approaches: redundancy and weak guidance. Modalities often contain "private" noise that is irrelevant to the global clustering task. Furthermore, many models fail to utilize the global clustering results to refine individual modality representations, missing the opportunity for the global structure to "guide" local feature learning.

The MSDIB Framework

The proposed MSDIB framework utilizes the Information Bottleneck (IB) principle to find a representation that balances compression and preservation. The goal is to create a latent space that is "cluster-friendly" by stripping away irrelevant details while maintaining the core signals shared across modalities.

The MSDIB framework architecture, highlighting the Information Compression and Information Preservation modules and the multi-level self-guidance mechanism.

As illustrated in the architecture, the framework is divided into two core components:

  1. Information Compression (IC): This module employs deep encoders to extract compact representations. It implements mutual compression by minimizing mutual information between modalities, thereby stripping away redundant, modality-specific data that does not contribute to the shared cluster structure.
  2. Information Preservation (IP): This module ensures the retention of critical shared signals by maximizing mutual information between private features and a fused global representation.

A key innovation of MSDIB is its multi-level self-guidance mechanism. This includes mutual guidance between modalities, guidance from fused global features to private features, and guidance from global cluster assignments (pseudo-labels) to local assignments. This creates a robust feedback loop that ensures all parts of the model converge toward a cohesive clustering result.

Experimental Evaluation

The model was evaluated on five widely recognized multi-modal datasets: Caltech-2V, ESP-Game, IAPR, MIRFlickr, and NUS-Wide. These datasets provide a range of image-text pairs and multi-feature image data, offering a comprehensive test of the framework's versatility across different data types and scales.

Key Results and Discussion

MSDIB consistently outperformed state-of-the-art (SOTA) methods, such as MFLVC and DIVICE, across all five datasets in terms of Accuracy (ACC) and Normalized Mutual Information (NMI). A notable highlight was the ESP-Game dataset, where MSDIB achieved a 14.1% improvement in Accuracy over the next best-performing method.

Parameter sensitivity analysis across five datasets, demonstrating the robustness of MSDIB to hyperparameters α and β.

Beyond raw performance, the framework demonstrated significant stability. Parameter sensitivity analysis indicates that MSDIB is relatively insensitive to its trade-off hyperparameters ($\alpha$ and $\beta$). As shown in the 3D visualizations above, the model maintains high accuracy across a wide range of settings, reducing the need for exhaustive hyperparameter tuning in practical applications.

Conclusion

MSDIB offers a theoretically grounded and empirically robust solution for multi-modal clustering. By integrating deep learning with Information Bottleneck theory, it successfully balances noise reduction with feature preservation. While the current framework is designed for aligned data, future work will focus on extending MSDIB to handle incomplete or unaligned multi-modal datasets, which are common in messy, real-world environments.

Summary updated

Abstract

Deep multi-modal clustering can extract useful information among modals, thus benefiting the final clustering and many related fields. However, existing multi-modal clustering methods have two major limitations. First, they often ignore different levels of guiding information from both the feature representations and cluster assignments, which thus are difficult in learning discriminative representations. Second, most methods fail to effectively eliminate redundant information between multi-modal data, negatively affecting clustering results. In this paper, we propose a novel multi-aspect self-guided deep information bottleneck (MSDIB) method for multi-modal clustering, which can effectively employ different aspects of guiding information for learning cluster-friendly information among modals. MSDIB mainly contains two parts: information compression and information preservation. In information compression, we extract from the private information of each modality to obtain the compact representation and meanwhile conduct mutual compression between them. In information preservation, the aim is to preserve the shared information among modals and the self-supervised information from the clustering results in each iteration. In the above process, there are mainly three aspects of self-guiding information, the modality-private information, the modality-shared information and the self-supervised pseudo label information. By minimizing the mutual information based objective function with a variational optimization method, we can fully extract useful discriminative information while eliminating the irrelevant parts. Extensive experimental results demonstrate that our method outperforms state-of-the-art multi-modal clustering methods, showcasing its superior performance and broad application prospects.
BibTeX
@inproceedings{hu2025multi,
  author = {Shizhe Hu and Jiahao Fan and Guoliang Zou and Yangdong Ye},
  title = {{Multi-aspect Self-guided Deep Information Bottleneck for Multi-modal Clustering}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i16.33903},
}
Figure