Generalized Multiview Shared Subspace Learning Using View Bootstrapping

Krishna Somandepalli, Shrikanth NarayananIEEE TSP 2021 · OpenReview

Abstract

A key objective in multiview learning is to model information common to multiple parallel views to improve downstream tasks. This work addresses scalability and view-agnosticism by introducing a neural method based on multiview correlation that captures shared information by subsampling views during training. We provide a theoretical upper bound on the number of views to subsample ($m \leq \sqrt{d}$) and demonstrate robustness across audio and visual tasks, including 3D object classification and pose-invariant face recognition.

Introduction

Multiview learning aims to extract a shared subspace capturing core information common across different data streams, such as multiple camera angles or audio channels. Traditional approaches face two primary challenges:

  1. Scalability: Models often require unique sub-networks for every view, leading to parameter explosion as the number of views increases.
  2. View-Agnosticism: Standard methods typically require explicit metadata about view identities, which is often unavailable in real-world scenarios.

This paper introduces view bootstrapping, a scalable, view-agnostic approach capable of handling hundreds of perspectives by randomly subsampling a small subset of views during training.

Methodology: View Bootstrapping

The core of the proposed method is the multi-view correlation (mv-corr) objective, which maximizes the ratio of "between-view" covariance (shared signal) to "within-view" covariance (noise).

Overview of the view bootstrapping architecture.

Instead of maintaining $M$ distinct sub-networks for $M$ views, the architecture uses a fixed number $m$ of identical sub-network branches. During each training iteration, $m$ views are randomly sampled from the total set with replacement. This "bootstrapping" approach ensures:

  • Identical Architectures: Sub-networks share the same structure but maintain separate weights, reducing overall model complexity.
  • View-Agnostic Training: Random sampling prevents the model from relying on specific view identities, forcing it to learn features invariant to the perspective.
  • Theoretical Efficiency: Using matrix concentration theory, the authors prove that to learn a $d$-dimensional embedding, one only needs to subsample $m \approx \sqrt{d}$ views.

Experimental Setup

The method was evaluated across three diverse domains:

  • 3D Object Classification (ModelNet): Using 2D renders from 12 and 80 camera angles.
  • Pose-Invariant Face Recognition (Multi-PIE): Images of 337 subjects across 15 poses and various lighting conditions.
  • Spoken Word Recognition (Speech Commands): Treating different speakers as distinct "views" of the same command.

Results and Discussion

The experiments confirmed the theoretical prediction regarding the number of subsampled views. As shown in Figure 2, increasing $m$ beyond $\sqrt{d}$ results in diminishing returns for clustering accuracy.

Clustering accuracy vs. bootstrap sample size m, validating the theoretical bound.

Key performance highlights include:

  • Generalization: On the 80-view ModelNet dataset, the method achieved 85.7% accuracy on unseen views, outperforming a supervised CNN baseline (80.3%).
  • Robustness to Pose: In face recognition, the model reached 93.3% average accuracy, significantly exceeding Deep CCA (69.8%).
  • Domain Adaptation: In speech recognition, the method reached 92.4% accuracy, surpassing state-of-the-art domain adversarial methods like DAN (77.9%).

The "bootstrapping" process acts as a powerful regularizer, compelling the network to identify truly shared structures across views. Furthermore, the authors optimized the covariance estimation to $O(M)$ time, ensuring the method scales linearly with the total number of views.

Conclusion

View bootstrapping provides a robust framework for learning from high-dimensional multiview data without requiring view-specific metadata. While the method effectively discards view-specific noise, future work could explore ways to retain unique modality-specific information for tasks where such details are beneficial. The current results underscore the applicability of this approach for large-scale, view-agnostic learning in the wild.

Summary updated

Abstract

A key objective in multiview learning is to model the information common to multiple parallel views of a class of objects/events to improve downstream tasks such as classification and clustering. In this context, two open research challenges remain; achieving scalability: how can we incorporate information from hundreds of views per event into a model? and being view-agnostic: how to learn robust multiview representations without knowledge of how these views are acquired? In this work, we study a neural method based on multiview correlation to capture the information shared across a large number of views by subsampling them in a view-agnostic manner during training. We analyze the error of this bootstrapped multiview correlation objective using matrix concentration theory to provide an upper bound on the number of views to subsample for a given embedding dimension. Our experiments on a diverse set of audio and visual tasks-multi-channel acoustic activity classification, spoken word recognition, 3D object classification, and pose-invariant face recognition-demonstrate the robustness of view bootstrapping to model a large number of views. Results and analysis underscore the applicability of our method for a view-agnostic learning setting.
BibTeX
@article{somandepalli2021generalized,
  author = {Krishna Somandepalli and Shrikanth Narayanan},
  title = {{Generalized Multiview Shared Subspace Learning Using View Bootstrapping}},
  journal = {IEEE Trans. Signal Process.},
  year = {2021},
  url = {https://openreview.net/forum?id=8I6bRiAYb2A},
}
Figure