Abstract
Standard methods for video recognition use large CNNs designed to capture spatio-temporal data. However, training these models requires a large amount of labeled training data, containing a wide variety of actions, scenes, settings and camera viewpoints. In this paper, we show that current convolutional neural network models are unable to recognize actions from camera viewpoints not present in their training data (i.e., unseen view action recognition). To address this, we develop approaches based on 3D representations and introduce a new geometric convolutional layer that can learn viewpoint invariant representations. Further, we introduce a new, challenging dataset for unseen view recognition and show the approaches ability to learn viewpoint invariant representations.
Introduction
Standard Convolutional Neural Networks (CNNs) are remarkably good at recognizing actions in videos, provided the test data looks like the training data. However, they are notoriously "view-dependent." If a model is trained on broadcast footage of a baseball game (a static, high-angle view) and then tested on a cell phone video taken from the bleachers, it often fails. While humans can easily recognize a "pitch" regardless of the angle, CNNs struggle because they don't inherently understand the 3D geometry of the world. This paper addresses unseen viewpoint activity recognition: how can we make a model recognize the same action from a completely new camera angle?
Methodology: The Neural Projection Layer
The authors propose moving beyond simple 2D image processing by incorporating geometric inductive biases directly into the network architecture through a novel Neural Projection Layer (NPL). Instead of just tracking joints (which can be error-prone and ignore objects), the NPL takes a 2D feature map and lifts it into a latent 3D world representation.
The pipeline functions as follows:
- Feature Lifting: The model estimates a 3D coordinate for every feature point.
- Camera Estimation: It learns to estimate the camera's rotation ($R$) and translation ($t$) for that specific video.
- Latent 3D Representation: Features are transformed into a viewpoint-invariant 3D world representation.
- Multi-view 2D Projections: The 3D data is projected into several "virtual" 2D views, which are stacked and fed into a final classifier.
To train this without 3D labels, a self-supervised geometric loss forces the model to produce the same 3D world representation for different videos of the same action, regardless of the original camera angle.
Datasets and Benchmarks
The approach was evaluated on several multi-view datasets, including Human3.6M, Toyota SmartHome (TSH), and NTU-RGB-D. To further test generalization, the authors introduced an extended MLB-YouTube benchmark. This dataset takes original broadcast views and adds 500 new YouTube videos of the same actions (swing, pitch, etc.) taken from diverse, "unseen" viewpoints like stands or batting cages.
Key Results and Insights
The NPL approach consistently outperformed standard CNNs and previous state-of-the-art methods on unseen views:
- MLB-YouTube: On unseen views, a standard ResNet-50 (pre-trained on Kinetics) achieved 30.2% accuracy. The NPL + Multi-view model boosted this to 42.7%.
- Human3.6M: In a cross-view setup where standard CNNs dropped to near-random levels (~9%), the proposed method maintained 87.5% accuracy.
- Latent 3D vs. Skeletons: On datasets like Toyota SmartHome, using only 3D pose (skeletons) actually hurt performance. The NPL captures the features of the objects in 3D space, which is critical for distinguishing actions like "picking up a cup" vs. "picking up a plate."
- Efficiency: The NPL is lightweight, adding only about 280k parameters (a ~1% increase for a ResNet-50) and adding only ~15ms of latency per 3-second video clip.
Conclusion
By incorporating a geometric structure like the NPL, models can learn true viewpoint invariance that massive pre-training alone cannot provide. While challenges remain regarding occlusions and noisy pose estimation in crowded environments, the NPL offers a robust path forward for "in-the-wild" video recognition from any angle.
Summary updated