Abstract
A key objective in multiview learning is to model information common to multiple parallel views to improve downstream tasks. This work addresses scalability and view-agnosticism by introducing a neural method based on multiview correlation that captures shared information by subsampling views during training. We provide a theoretical upper bound on the number of views to subsample ($m \leq \sqrt{d}$) and demonstrate robustness across audio and visual tasks, including 3D object classification and pose-invariant face recognition.
Introduction
Multiview learning aims to extract a shared subspace capturing core information common across different data streams, such as multiple camera angles or audio channels. Traditional approaches face two primary challenges:
- Scalability: Models often require unique sub-networks for every view, leading to parameter explosion as the number of views increases.
- View-Agnosticism: Standard methods typically require explicit metadata about view identities, which is often unavailable in real-world scenarios.
This paper introduces view bootstrapping, a scalable, view-agnostic approach capable of handling hundreds of perspectives by randomly subsampling a small subset of views during training.
Methodology: View Bootstrapping
The core of the proposed method is the multi-view correlation (mv-corr) objective, which maximizes the ratio of "between-view" covariance (shared signal) to "within-view" covariance (noise).
Instead of maintaining $M$ distinct sub-networks for $M$ views, the architecture uses a fixed number $m$ of identical sub-network branches. During each training iteration, $m$ views are randomly sampled from the total set with replacement. This "bootstrapping" approach ensures:
- Identical Architectures: Sub-networks share the same structure but maintain separate weights, reducing overall model complexity.
- View-Agnostic Training: Random sampling prevents the model from relying on specific view identities, forcing it to learn features invariant to the perspective.
- Theoretical Efficiency: Using matrix concentration theory, the authors prove that to learn a $d$-dimensional embedding, one only needs to subsample $m \approx \sqrt{d}$ views.
Experimental Setup
The method was evaluated across three diverse domains:
- 3D Object Classification (ModelNet): Using 2D renders from 12 and 80 camera angles.
- Pose-Invariant Face Recognition (Multi-PIE): Images of 337 subjects across 15 poses and various lighting conditions.
- Spoken Word Recognition (Speech Commands): Treating different speakers as distinct "views" of the same command.
Results and Discussion
The experiments confirmed the theoretical prediction regarding the number of subsampled views. As shown in Figure 2, increasing $m$ beyond $\sqrt{d}$ results in diminishing returns for clustering accuracy.
Key performance highlights include:
- Generalization: On the 80-view ModelNet dataset, the method achieved 85.7% accuracy on unseen views, outperforming a supervised CNN baseline (80.3%).
- Robustness to Pose: In face recognition, the model reached 93.3% average accuracy, significantly exceeding Deep CCA (69.8%).
- Domain Adaptation: In speech recognition, the method reached 92.4% accuracy, surpassing state-of-the-art domain adversarial methods like DAN (77.9%).
The "bootstrapping" process acts as a powerful regularizer, compelling the network to identify truly shared structures across views. Furthermore, the authors optimized the covariance estimation to $O(M)$ time, ensuring the method scales linearly with the total number of views.
Conclusion
View bootstrapping provides a robust framework for learning from high-dimensional multiview data without requiring view-specific metadata. While the method effectively discards view-specific noise, future work could explore ways to retain unique modality-specific information for tasks where such details are beneficial. The current results underscore the applicability of this approach for large-scale, view-agnostic learning in the wild.
Summary updated