Incomplete Multi-Modal Visual Data Grouping

Handong Zhao, Hongfu Liu, Yun Fu

Abstract

Multi-modal data clustering is a critical task in computer vision, yet real-world scenarios often suffer from incomplete observations where specific modalities are missing for certain samples. This paper presents a novel unsupervised framework for Incomplete Multi-modality Grouping (IMG). By integrating latent subspace learning with a complete graph Laplacian term, the proposed method preserves the global structure of heterogeneous data. Experimental results across multiple datasets demonstrate that this approach significantly outperforms existing state-of-the-art methods, particularly in scenarios with high data incompleteness.

Introduction

In the contemporary landscape of machine learning, data is frequently harvested from diverse sources or modalities. For instance, a surveillance system might capture both RGB video and depth information. While this multi-modal approach provides a comprehensive view of the environment, it is often plagued by the "missing modality" problem. Sensors can fail due to environmental factors, or certain data types might be restricted for specific samples, leading to incomplete datasets.

Traditional multi-modal clustering algorithms generally operate under the assumption of completeness—that every sample possesses observations across all modalities. When faced with missing data, these methods typically resort to suboptimal strategies, such as discarding incomplete samples or imputing missing values with averages. The former leads to significant information loss, while the latter introduces noise that can degrade clustering performance. This paper addresses the challenge of grouping heterogeneous, incomplete samples into coherent clusters by leveraging the underlying global relationships between all data points.

Proposed Methodology: Incomplete Multi-modality Grouping

The core of the proposed framework is a unified objective function designed to handle data incompleteness gracefully. The method performs two primary tasks simultaneously: Latent Subspace Projection and the construction of a Complete Graph Laplacian.

Latent Subspace Projection

The framework transforms original, incomplete data from various modalities into a shared, complete $k$-dimensional latent space. This transformation is achieved through matrix factorization techniques. Unlike standard Non-negative Matrix Factorization (NMF), the authors relax the non-negativity constraint to simplify the optimization process while still effectively capturing the essential features of the multi-modal data.

Conceptual framework for Incomplete Multi-modality Grouping (IMG), illustrating the projection of incomplete RGB and Depth modalities into a shared latent space for grouping.

Complete Graph Laplacian

The most significant innovation is the introduction of a "complete graph Laplacian" term. This term measures the affinity (similarity) between all samples within the learned latent space, regardless of whether a sample is complete or partial. By applying a graph Laplacian to the entire dataset, the model creates a structural "bridge" between complete-modal samples and partial-modal samples. This ensures that the learned latent space is not merely a projection but a structure-preserving representation where even samples with missing modalities are correctly positioned relative to their complete counterparts.

Optimization Strategy

The resulting objective function is non-convex, presenting a significant optimization challenge. To address this, the authors employ the Augmented Lagrange Multiplier (ALM) method. By introducing auxiliary variables, the algorithm can iteratively update the global affinity matrix and the latent representations, converging toward a stable solution that balances data reconstruction and structural preservation.

Experimental Evaluation

The effectiveness of the IMG framework was evaluated on five diverse datasets, representing various challenges in visual data grouping:

  1. Synthetic Data: A two-component Gaussian mixture model used for controlled testing.
  2. MSR Action Pairs: An RGB-D action dataset featuring 12 activity types.
  3. MSR Daily Activity: A complex RGB-D dataset with 16 daily activity types.
  4. BUAA NirVis: A face database containing visual spectral (VIS) and near-infrared (NIR) images.
  5. UCI Handwritten Digits: A dataset of digits 0-9 represented by Fourier coefficients and pixel averages.

Results and Discussion

The proposed method was compared against several baseline approaches, including Best Single View (BSV), Feature Concatenation (Concat), and Partial Multi-View Clustering (PVC).

Superiority in Data Scarcity

The IMG framework consistently outperformed competitors across all metrics. On synthetic data, it achieved a Normalized Mutual Information (NMI) improvement of approximately 31.83%. In real-world datasets, the NMI gains reached up to 10.34% over state-of-the-art baselines.

Robustness to Incompleteness

A key finding of the study is the method's robustness as the Partial Example Ratio (PER) increases. While the performance of traditional methods drops sharply when a large percentage of data is missing, the IMG method remains remarkably stable. For example, on the MSR Daily Activity dataset with a 90% missing data rate, the proposed method still maintained significantly higher NMI than the next best competitor.

Performance comparison (NMI) across different Partial Example Ratios (PER), demonstrating the robustness of the proposed method compared to baselines.

Conclusion and Future Work

This paper introduces a robust framework for grouping incomplete multi-modal data. By leveraging a complete graph Laplacian in a shared latent space, the model effectively preserves global data structures even under high levels of incompleteness.

Future research directions include exploring more sophisticated matrix factorization techniques, such as weighted NMF or $\ell_{2,1}$ regularizers, to further enhance robustness. Additionally, addressing the computational complexity of the graph inverse operation—currently $O(N^3)$—will be essential for scaling the method to extremely large-scale datasets. Finally, extending the formulation to handle scenarios with a high number of heterogeneous modalities remains an open area for exploration.

Summary updated

Abstract

Nowadays multi-modal visual data are much easier to access as the technology develops. Nevertheless, there is an underlying problem hidden behind the emerging multi-modality techniques: What if one/more modal data fail? Motivated by this question, we propose an unsupervised method which well handles the incomplete multimodal data by transforming the original and incomplete data to a new and complete representation in a latent space. Different from the existing efforts that simply project data from each modality into a common subspace, a novel graph Laplacian term with a good probabilistic interpretation is proposed to couple the incomplete multi-modal samples. In such a way, a compact global structure over the entire heterogeneous data is well preserved, leading to a strong grouping discriminability. As a non-trivial contribution, we provide the optimization solution to the proposed model. In experiments, we extensively test our method and competitors on one synthetic data, two RGB-D video datasets and two image datasets. The superior results validate the benefits of the proposed method, especially when multimodal data suffer from large incompleteness.
BibTeX
@inproceedings{zhao2016incomplete,
  author = {Handong Zhao and Hongfu Liu and Yun Fu},
  title = {{Incomplete Multi-Modal Visual Data Grouping}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2016},
}
Figure