Imagine trying to understand a complex concept using both an image and its text caption. Each source of information—or "view"—provides a distinct perspective. A naive approach to combining them might simply concatenate their feature vectors. In practice, however, this fails: different views have completely different dimensionalities, scales, and noise distributions. Simple concatenation treats view-specific noise as a meaningful signal and completely ignores the underlying correlation between the modalities.
To solve this, researchers use shared latent subspace learning. Instead of forcing disparate features together in their raw forms, this approach projects all views into a single, lower-dimensional, shared latent space. This space acts as a mathematical consensus mechanism. By forcing different modalities to map to the same underlying coordinate system, the models filter out view-specific noise and uncover the true, shared semantic meaning.
Joint Matrix Factorization: Enforcing a Shared Basis
How do we mathematically force multiple distinct datasets to agree on a single representation? One of the most direct and elegant ways is through joint matrix factorization.
If we have multiple data matrices (one for each view), we can decompose them simultaneously. As demonstrated by Zhenfeng Zhu et al. in Shared Subspace Learning for Latent Representation of Multi-View Data, we can minimize the reconstruction error between the original feature spaces and a shared subspace.
$$\mathbf{X}^{(v)} \approx \mathbf{W}^{(v)} \mathbf{H}$$
In this formulation, $\mathbf{X}^{(v)}$ represents the data matrix for view $v$. The key trick is that while the dictionary or basis matrix $\mathbf{W}^{(v)}$ is view-specific to handle different dimensionalities, the latent coefficient matrix $\mathbf{H}$ is strictly constrained to be identical across all views. This constraint forces the optimization process to find a single, consistent representation that explains the data across all observation methods simultaneously.
To capture more complex, non-linear structures, Handong Zhao et al. extended this concept to deep architectures in Multi-View Clustering via Deep Matrix Factorization. By stacking semi-nonnegative matrix factorization layers, their framework leverages hierarchical abstractions to align disparate feature sets into a single, highly discriminative space.
Generative Reconstruction and Information Bottlenecks
In deep learning, this consensus is often enforced using encoder-decoder architectures or Variational Autoencoders (VAEs). Instead of just aligning projections, the network is forced to compress all views into a central latent vector that contains enough information to reconstruct all original views simultaneously.
For instance, in Shared Generative Latent Representation Learning for Multi-View Clustering, Ming Yin et al. utilize separate view-specific encoders that feed into a central fusion module, which then uses multiple decoders to reconstruct every original view from that single latent point.
This architecture acts as a powerful information bottleneck. Because the central embedding has a limited capacity, it cannot afford to waste bits on view-specific noise (which is unpredictable from the other views). Instead, it is forced to preserve only the core, complementary information required to satisfy all decoders at once. Shaika Chowdhury et al. successfully applied this joint reconstruction objective in Med2Meta: Learning Representations of Medical Concepts with Meta-Embeddings to compress and preserve salient information across highly heterogeneous Electronic Health Record streams, such as clinical notes and lab results.
Disentangling Shared and View-Specific Features
A potential pitfall of forcing a single, unified representation is that we might accidentally discard useful information that happens to be unique to a single view. For example, in video analysis, motion features are completely absent from static image frames, yet they are crucial for understanding actions.
To prevent the shared space from choking out these unique signals, advanced frameworks explicitly decompose the data into two distinct components: a shared latent factor (the common cause) and a view-specific factor (the unique variation).
Zhuowei Cai et al. pioneered this representational strategy in Multi-view Super Vector for Action Recognition. By using a Mixture of Probabilistic Canonical Correlation Analyzers, they decomposed pairs of local descriptors into shared latent factors and view-specific gradient vectors.
Zhibin Wan formalized this balance from an information-theoretic perspective in Multi-View Information-Bottleneck Representation Learning. The proposed CMIB-Nets isolate shared cross-view structures from unique intrinsic features, optimizing a variational bound that ensures the final fused representation achieves the perfect equilibrium between cross-view consistency and view-specific complementarity.
Overcoming Real-World Imperfections
Shared latent spaces also provide elegant solutions to two major real-world data challenges: missing data and computational scalability.
Incomplete Views
In practical applications, it is common for certain views to be missing entirely for some samples (e.g., a patient who has lab results but no clinical notes). Because the shared latent space is anchored by the samples that do possess all views, it acts as an information bridge. In Incomplete Multi-view Clustering via Graph Regularized Matrix Factorization, Jie Wen et al. showed that by enforcing consistency across latent codes during joint factorization, the model can seamlessly transfer structural knowledge from complete samples to incomplete ones, effectively completing the underlying graph structure.
The Scalability Bottleneck
If a dataset has $V$ different views, performing exhaustive pairwise alignments between them requires $O(V^2)$ operations, which quickly becomes computationally intractable. To bypass this bottleneck, Xiangzhu Meng et al. introduced a centroid-based integration strategy in The Similarity-Consensus Regularized Multi-view Learning for Dimension Reduction. Instead of comparing views to each other, every individual view is forced to converge toward a single, shared "centroid" representation. This reduces the optimization complexity to $O(V)$, providing a highly scalable common ground for multi-source data integration.
Key Foundational Work
Multi-view Super Vector for Action Recognition (2014) — contributed 2 concept elements to this direction.
Shared Subspace Learning for Latent Representation of Multi-View Data (2014) — contributed 1 concept elements to this direction.
NMF-KNN: Image Annotation Using Weighted Multi-view Non-negative Matrix Factorization (2014) — contributed 1 concept elements to this direction.