Factorized Orthogonal Latent Spaces

Mathieu Salzmann, Carl Henrik Ek, Raquel Urtasun, Trevor DarrellAISTATS 2010 · OpenReview

Abstract: Existing approaches to multi-view learning are particularly effective when the views are either independent (i.e., multi-kernel approaches) or fully dependent (i.e., shared latent spaces). However, real-world data often contains a mixture of shared and private information. This paper introduces Factorized Orthogonal Latent Spaces (FOLS), a robust framework that untangles multi-view data into shared and private latent components. By employing orthogonality and rank constraints, FOLS automatically discovers the optimal dimensionality of these spaces, effectively handling correlated noise and improving performance in complex tasks such as 3D human pose estimation.

Introduction

In multi-view learning, the goal is to combine information from multiple data streams—such as audio and video or different sensor modalities—to improve predictive performance. Traditional algorithms typically assume one of two extremes: either the views are entirely independent or they are fully dependent, mapping to a single "shared" latent space.

Real-world data is rarely so clean. Views typically share some information while retaining private, view-specific details. Forcing these distinct signals into a single shared space often leads to "private" noise contaminating the shared signal, degrading model accuracy. While some recent methods have attempted to factorize latent spaces into shared and private components, they remain sensitive to initialization, struggle with correlated noise, and require manual tuning of latent dimensions.

Methodology

The authors propose Factorized Orthogonal Latent Spaces (FOLS), a regularization framework designed to cleanly separate shared and private information. The FOLS objective function introduces three critical constraints:

  1. Orthogonality: It penalizes the inner product between shared and private latent spaces. This ensures the model does not redundantly encode the same information across different spaces.
  2. Low Dimensionality (Rank Constraint): FOLS uses a logarithmic penalty on the singular values of the latent spaces. This encourages sparsity, automatically shrinking irrelevant dimensions to zero and discovering the intrinsic dimensionality of the data without expensive cross-validation.
  3. Energy Conservation: To prevent the model from trivially shrinking all coordinates to zero, a term is added to ensure the energy of the latent spectrum matches the original observation space.
The FOLS graphical model illustrates the separation of shared latent space (X) and private latent spaces (Z) for multiple views (Y).

This framework was integrated into two existing models: the directed shared Gaussian Process Latent Variable Model (sGPLVM) and the undirected shared Kernel Information Embedding (sKIE).

Experimental Results

Synthetic Validation

The framework was first tested on a synthetic dataset of sinusoidal signals with highly correlated noise. Traditional methods like Canonical Correlation Analysis (CCA) failed, mistakenly grouping correlated noise into the shared space. In contrast, FOLS successfully isolated the true generative signals from the noise.

Synthetic results (a-f) demonstrate the recovery of generative signals and the analysis of posterior variance, highlighting how FOLS handles multi-modal data.

3D Human Pose Estimation

The models were further evaluated on the HumanEva Dataset, a benchmark for 3D human pose estimation from monocular images. This task is inherently multi-modal; a single 2D silhouette can correspond to multiple valid 3D poses (e.g., ambiguity regarding which leg is forward).

Key findings included:

  • Superior Accuracy: FOLS models significantly outperformed standard sGPLVM, sKIE, and previous state-of-the-art factorization techniques.
  • Automatic Pruning: The rank constraint successfully pruned the latent spaces to an optimal dimensionality of 5, capturing 95% of the variance without manual intervention.
  • Robustness: By factorizing out private information not present in the 2D image, FOLS prevented the shared space from becoming tangled, leading to more stable pose recovery.

Conclusion and Future Work

FOLS provides a principled way to handle the complexity of multi-view data by enforcing orthogonality and sparsity. While the rank constraint introduces some computational overhead ($O(N^3)$), the authors mitigate this through iterative pruning of small singular values. Future work includes applying FOLS to multi-agent modeling and extending the framework to handle missing data in semi-supervised learning environments.

Summary updated

Abstract

Existing approaches to multi-view learning are particularly effective when the views are either independent (i.e, multi-kernel approaches) or fully dependent (i.e., shared latent spaces). However, ...
BibTeX
@inproceedings{salzmann2010factorized,
  author = {Mathieu Salzmann and Carl Henrik Ek and Raquel Urtasun and Trevor Darrell},
  title = {{Factorized Orthogonal Latent Spaces}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2010},
  url = {https://openreview.net/forum?id=KZHMrCEQ_nW},
}
Figure