Abstract
Multi-modal data clustering is a critical task in computer vision, yet real-world scenarios often suffer from incomplete observations where specific modalities are missing for certain samples. This paper presents a novel unsupervised framework for Incomplete Multi-modality Grouping (IMG). By integrating latent subspace learning with a complete graph Laplacian term, the proposed method preserves the global structure of heterogeneous data. Experimental results across multiple datasets demonstrate that this approach significantly outperforms existing state-of-the-art methods, particularly in scenarios with high data incompleteness.
Introduction
In the contemporary landscape of machine learning, data is frequently harvested from diverse sources or modalities. For instance, a surveillance system might capture both RGB video and depth information. While this multi-modal approach provides a comprehensive view of the environment, it is often plagued by the "missing modality" problem. Sensors can fail due to environmental factors, or certain data types might be restricted for specific samples, leading to incomplete datasets.
Traditional multi-modal clustering algorithms generally operate under the assumption of completeness—that every sample possesses observations across all modalities. When faced with missing data, these methods typically resort to suboptimal strategies, such as discarding incomplete samples or imputing missing values with averages. The former leads to significant information loss, while the latter introduces noise that can degrade clustering performance. This paper addresses the challenge of grouping heterogeneous, incomplete samples into coherent clusters by leveraging the underlying global relationships between all data points.
Proposed Methodology: Incomplete Multi-modality Grouping
The core of the proposed framework is a unified objective function designed to handle data incompleteness gracefully. The method performs two primary tasks simultaneously: Latent Subspace Projection and the construction of a Complete Graph Laplacian.
Latent Subspace Projection
The framework transforms original, incomplete data from various modalities into a shared, complete $k$-dimensional latent space. This transformation is achieved through matrix factorization techniques. Unlike standard Non-negative Matrix Factorization (NMF), the authors relax the non-negativity constraint to simplify the optimization process while still effectively capturing the essential features of the multi-modal data.
Complete Graph Laplacian
The most significant innovation is the introduction of a "complete graph Laplacian" term. This term measures the affinity (similarity) between all samples within the learned latent space, regardless of whether a sample is complete or partial. By applying a graph Laplacian to the entire dataset, the model creates a structural "bridge" between complete-modal samples and partial-modal samples. This ensures that the learned latent space is not merely a projection but a structure-preserving representation where even samples with missing modalities are correctly positioned relative to their complete counterparts.
Optimization Strategy
The resulting objective function is non-convex, presenting a significant optimization challenge. To address this, the authors employ the Augmented Lagrange Multiplier (ALM) method. By introducing auxiliary variables, the algorithm can iteratively update the global affinity matrix and the latent representations, converging toward a stable solution that balances data reconstruction and structural preservation.
Experimental Evaluation
The effectiveness of the IMG framework was evaluated on five diverse datasets, representing various challenges in visual data grouping:
- Synthetic Data: A two-component Gaussian mixture model used for controlled testing.
- MSR Action Pairs: An RGB-D action dataset featuring 12 activity types.
- MSR Daily Activity: A complex RGB-D dataset with 16 daily activity types.
- BUAA NirVis: A face database containing visual spectral (VIS) and near-infrared (NIR) images.
- UCI Handwritten Digits: A dataset of digits 0-9 represented by Fourier coefficients and pixel averages.
Results and Discussion
The proposed method was compared against several baseline approaches, including Best Single View (BSV), Feature Concatenation (Concat), and Partial Multi-View Clustering (PVC).
Superiority in Data Scarcity
The IMG framework consistently outperformed competitors across all metrics. On synthetic data, it achieved a Normalized Mutual Information (NMI) improvement of approximately 31.83%. In real-world datasets, the NMI gains reached up to 10.34% over state-of-the-art baselines.
Robustness to Incompleteness
A key finding of the study is the method's robustness as the Partial Example Ratio (PER) increases. While the performance of traditional methods drops sharply when a large percentage of data is missing, the IMG method remains remarkably stable. For example, on the MSR Daily Activity dataset with a 90% missing data rate, the proposed method still maintained significantly higher NMI than the next best competitor.
Conclusion and Future Work
This paper introduces a robust framework for grouping incomplete multi-modal data. By leveraging a complete graph Laplacian in a shared latent space, the model effectively preserves global data structures even under high levels of incompleteness.
Future research directions include exploring more sophisticated matrix factorization techniques, such as weighted NMF or $\ell_{2,1}$ regularizers, to further enhance robustness. Additionally, addressing the computational complexity of the graph inverse operation—currently $O(N^3)$—will be essential for scaling the method to extremely large-scale datasets. Finally, extending the formulation to handle scenarios with a high number of heterogeneous modalities remains an open area for exploration.
Summary updated