Robust Multi-View Representation Learning (Student Abstract)

Sibi Venkatesan, James Kyle Miller, Artur Dubrawski

Introduction

In complex domains such as autonomous driving and medical diagnostics, data is often collected from multiple heterogeneous sources or "views"—for instance, combining LiDAR, radar, and camera feeds. Traditional multi-view representation learning, exemplified by Canonical Correlation Analysis (CCA), typically seeks a global structure shared across all views. However, this approach often fails to capture "local" dependencies where only a subset of sensors are correlated. This paper proposes two novel frameworks, MOCCA (Multi-view One-vs-Rest CCA) and RMVAE (Robust Multi-view AutoEncoder), designed to capture these local relationships and improve robustness against missing or noisy data.

Proposed Methods

MOCCA: Multi-view One-vs-Rest CCA

To avoid the exponential complexity of modeling every possible subset of views, MOCCA reformulates the problem into $K$ "one-vs-rest" sub-problems. For each view $k$, the model attempts to predict its representation using a combination of the remaining $K-1$ views.

The core innovation is the application of a group-sparse penalty ($L_\infty$ norm) during the optimization process. This penalty encourages the model to select only the most informative subset of views to reconstruct the target view, effectively performing automatic view selection and uncovering the underlying redundancy structure of the data.

RMVAE: Robust Multi-view AutoEncoder

While MOCCA focuses on linear relationships, RMVAE extends these principles to non-linear representations using neural networks. RMVAE utilizes an AutoEncoder architecture where the latent space is forced to capture cross-view redundancies through view-level dropout. During training, entire views are randomly masked, requiring the encoder to learn a representation that can reconstruct all views even when some inputs are missing. This "stress testing" ensures the model does not become overly reliant on any single sensor.

Experimental Setup

The methods were evaluated using a synthetic four-view dataset ($X_1, X_2, X_3, X_4$) generated from four underlying feature sets ($A, B, C, D$). The views were constructed with specific redundancies (e.g., $X_1$ contains features $B, C, D$ while $X_2$ contains $A, C, D$), such that any two views provide sufficient information to reconstruct the others. This controlled environment allows for direct verification of whether the models can recover the known latent structure.

Results and Discussion

The effectiveness of the group-sparse penalty in MOCCA is clearly demonstrated through the visualization of the redundancy matrix.

Figure 1: Comparison of redundancy matrices in MOCCA. The left matrix, generated without group-sparsity, shows significant noise and blurred relationships. The right matrix, utilizing the group-sparse penalty, exhibits a sharp, blocky structure that accurately reflects the synthetic dataset's underlying dependencies.

As shown in Figure 1, the group-sparse penalty acts as a discovery tool, filtering out irrelevant correlations and highlighting the true local dependencies between views. Without this penalty, the relationship map is noisy and difficult to interpret; with it, the underlying block structure of the synthetic data is recovered.

For RMVAE, quantitative analysis confirmed that the model successfully learns to aggregate information. Reconstruction error decreased monotonically as more views were provided at test time, indicating that the latent representation effectively exploits the redundancies learned during the view-level dropout training phase.

Conclusion and Future Work

This work highlights the importance of modeling local view relationships for robust representation learning. By utilizing group-sparsity in MOCCA and view-level dropout in RMVAE, the authors provide computationally efficient methods for discovering and exploiting data redundancy. Future research directions include scaling MOCCA for high-dimensional datasets via greedy step-wise selection and extending RMVAE with flow-based generative models to handle more complex data distributions.

Summary updated

Abstract

Multi-view data has become ubiquitous, especially with multi-sensor systems like self-driving cars or medical patient-side monitors. We propose two methods to approach robust multi-view representation learning with the aim of leveraging local relationships between views.The first is an extension of Canonical Correlation Analysis (CCA) where we consider multiple one-vs-rest CCA problems, one for each view. We use a group-sparsity penalty to encourage finding local relationships. The second method is a straightforward extension of a multi-view AutoEncoder with view-level drop-out.We demonstrate the effectiveness of these methods in simple synthetic experiments. We also describe heuristics and extensions to improve and/or expand on these methods.
BibTeX
@inproceedings{venkatesan2020robust,
  author = {Sibi Venkatesan and James Kyle Miller and Artur Dubrawski},
  title = {{Robust Multi-View Representation Learning (Student Abstract)}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i10.7242},
}
Figure