Scaling Subspace-Driven Approaches Using Information Fusion

Sally Ghanem, Hamid KrimArtificial 2023 · DOI · OpenAlex

Abstract

In this work, we seek to exploit the deep structure of multi-modal data to robustly exploit the group subspace distribution of the information using the Convolutional Neural Networks (CNNs) formalism. Upon unfolding the set of subspaces constituting each data modality, and learning their corresponding encoders, an optimized integration of the generated inherent information is carried out to yield a characterization of various classes. Referred to as deep Multimodal Robust Group Subspace Clustering (DRoGSuRe), this approach is compared against the independently developed state-of-the-art approach named Deep Multimodal Subspace Clustering (DMSC). Experiments on different multimodal datasets show that our approach is competitive and more robust in the presence of noise.

Introduction

Multi-modal data processing is increasingly critical in artificial intelligence, where systems must integrate diverse sensory inputs—such as infrared, visible light, and polarimetric data—to achieve a comprehensive understanding of an environment. A significant challenge in this domain is the effective fusion of these "senses" without allowing noise or failure in a single modality to degrade the overall system performance.

Traditional subspace clustering methods rely on the Union of Subspaces (UoS) hypothesis, which posits that high-dimensional data points lie near a collection of low-dimensional subspaces. While effective for single-modality data, extending this to multi-modal contexts often leads to rigid architectures. For instance, Deep Multimodal Subspace Clustering (DMSC) typically forces all modalities into a shared latent space prematurely, making the system vulnerable to modality-specific noise.

The DRoGSuRe Framework

To address these limitations, the authors introduce DRoGSuRe (Deep Multimodal Robust Group Subspace Recovery). This framework utilizes a "parallel bank" architecture that preserves the unique characteristics of each modality while enforcing a shared underlying structure.

Overview of the DRoGSuRe architecture showing the parallel processing of multiple modalities from encoding to decoding.

The architecture consists of four primary components:

  1. Parallel CNN Encoders: Dedicated networks extract features from each modality independently.
  2. Independent Self-Expressive Layers: Each modality maintains its own self-representation matrix ($W$). By employing a group $l_{1,2}$-norm and a commutation loss ($W_1W_2 - W_2W_1 = 0$), the model ensures that different modalities align their structures (sharing eigenvectors) without losing modality-specific "private" information.
Detail of the independent self-expressive layers for multiple modalities 1 through T.
  1. Information Fusion via Aggregation: Instead of raw data fusion, DRoGSuRe aggregates the sparse coefficient matrices from each modality to construct a final Affinity Matrix.
  2. Spectral Clustering: The resulting affinity matrix is used to group data points into distinct clusters.

Robustness and Key Insights

A core innovation of DRoGSuRe is the use of commutation as a constraint. By forcing coefficient matrices to commute, the model mathematically ensures that while modalities may differ in their specific data distributions, they agree on the fundamental categorical boundaries. This approach provides "graceful degradation": as sensors fail or noise increases, the clustering accuracy declines predictably rather than catastrophically. The framework theoretically offers an $m$-improvement in resilience, where $m$ is the number of modalities.

Experimental Results

The approach was evaluated on the Extended Yale-B (EYB) and ARL Polarimetric Face datasets.

  • Accuracy: DRoGSuRe achieved 100% clustering accuracy on the ARL dataset under noiseless conditions, surpassing the DMSC baseline of 98.33%.
  • Noise Resilience: In scenarios where modalities were intentionally distorted (e.g., pixel shuffling), DRoGSuRe maintained high performance (approximately 95% accuracy with one distorted modality), whereas DMSC's performance dropped to roughly 87%.
  • Sensor Failure: The model demonstrated superior robustness during inference when entire modalities were missing, particularly on the complex ARL dataset.

Conclusion

DRoGSuRe represents a significant step forward in robust multi-modal subspace clustering. By allowing each modality its own degrees of freedom through parallel encoding and self-expression, the framework mitigates the risks associated with early-stage fusion. While the parallel architecture introduces additional computational overhead, the gains in noise resilience and accuracy make it a compelling choice for real-world sensor fusion tasks, such as autonomous perception and advanced biometric identification.

Summary updated

Abstract

In this work, we seek to exploit the deep structure of multi-modal data to robustly exploit the group subspace distribution of the information using the Convolutional Neural Networks (CNNs) formalism. Upon unfolding the set of subspaces constituting each data modality, and learning their corresponding encoders, an optimized integration of the generated inherent information is carried out to yield a characterization of various classes. Referred to as deep Multimodal Robust Group Subspace Clustering (DRoGSuRe), this approach is compared against the independently developed state-of-the-art approach named Deep Multimodal Subspace Clustering (DMSC). Experiments on different multimodal datasets show that our approach is competitive and more robust in the presence of noise.
BibTeX
@inproceedings{ghanem2023scaling,
  author = {Sally Ghanem and Hamid Krim},
  title = {{Scaling Subspace-Driven Approaches Using Information Fusion}},
  booktitle = {Artificial},
  year = {2023},
  doi = {10.5772/intechopen.109946},
}
Figure