Recognizing Actions in Videos from Unseen Viewpoints

A. J. Piergiovanni, Michael S. Ryoo

Abstract

Standard methods for video recognition use large CNNs designed to capture spatio-temporal data. However, training these models requires a large amount of labeled training data, containing a wide variety of actions, scenes, settings and camera viewpoints. In this paper, we show that current convolutional neural network models are unable to recognize actions from camera viewpoints not present in their training data (i.e., unseen view action recognition). To address this, we develop approaches based on 3D representations and introduce a new geometric convolutional layer that can learn viewpoint invariant representations. Further, we introduce a new, challenging dataset for unseen view recognition and show the approaches ability to learn viewpoint invariant representations.

Introduction

Standard Convolutional Neural Networks (CNNs) are remarkably good at recognizing actions in videos, provided the test data looks like the training data. However, they are notoriously "view-dependent." If a model is trained on broadcast footage of a baseball game (a static, high-angle view) and then tested on a cell phone video taken from the bleachers, it often fails. While humans can easily recognize a "pitch" regardless of the angle, CNNs struggle because they don't inherently understand the 3D geometry of the world. This paper addresses unseen viewpoint activity recognition: how can we make a model recognize the same action from a completely new camera angle?

Comparison of seen broadcast views vs. unseen amateur views in the MLB-YouTube dataset.

Methodology: The Neural Projection Layer

The authors propose moving beyond simple 2D image processing by incorporating geometric inductive biases directly into the network architecture through a novel Neural Projection Layer (NPL). Instead of just tracking joints (which can be error-prone and ignore objects), the NPL takes a 2D feature map and lifts it into a latent 3D world representation.

The pipeline functions as follows:

  1. Feature Lifting: The model estimates a 3D coordinate for every feature point.
  2. Camera Estimation: It learns to estimate the camera's rotation ($R$) and translation ($t$) for that specific video.
  3. Latent 3D Representation: Features are transformed into a viewpoint-invariant 3D world representation.
  4. Multi-view 2D Projections: The 3D data is projected into several "virtual" 2D views, which are stacked and fed into a final classifier.
Overview of the proposed video recognition pipeline with the Neural Projection Layer.

To train this without 3D labels, a self-supervised geometric loss forces the model to produce the same 3D world representation for different videos of the same action, regardless of the original camera angle.

Datasets and Benchmarks

The approach was evaluated on several multi-view datasets, including Human3.6M, Toyota SmartHome (TSH), and NTU-RGB-D. To further test generalization, the authors introduced an extended MLB-YouTube benchmark. This dataset takes original broadcast views and adds 500 new YouTube videos of the same actions (swing, pitch, etc.) taken from diverse, "unseen" viewpoints like stands or batting cages.

Key Results and Insights

The NPL approach consistently outperformed standard CNNs and previous state-of-the-art methods on unseen views:

  • MLB-YouTube: On unseen views, a standard ResNet-50 (pre-trained on Kinetics) achieved 30.2% accuracy. The NPL + Multi-view model boosted this to 42.7%.
  • Human3.6M: In a cross-view setup where standard CNNs dropped to near-random levels (~9%), the proposed method maintained 87.5% accuracy.
  • Latent 3D vs. Skeletons: On datasets like Toyota SmartHome, using only 3D pose (skeletons) actually hurt performance. The NPL captures the features of the objects in 3D space, which is critical for distinguishing actions like "picking up a cup" vs. "picking up a plate."
  • Efficiency: The NPL is lightweight, adding only about 280k parameters (a ~1% increase for a ResNet-50) and adding only ~15ms of latency per 3-second video clip.

Conclusion

By incorporating a geometric structure like the NPL, models can learn true viewpoint invariance that massive pre-training alone cannot provide. While challenges remain regarding occlusions and noisy pose estimation in crowded environments, the NPL offers a robust path forward for "in-the-wild" video recognition from any angle.

Summary updated

Abstract

Standard methods for video recognition use large CNNs designed to capture spatio-temporal data. However, training these models requires a large amount of labeled training data, containing a wide variety of actions, scenes, settings and camera viewpoints. In this paper, we show that current convolutional neural network models are unable to recognize actions from camera viewpoints not present in their training data (i.e., unseen view action recognition). To address this, we develop approaches based on 3D representations and introduce a new geometric convolutional layer that can learn viewpoint invariant representations. Further, we introduce a new, challenging dataset for unseen view recognition and show the approaches ability to learn viewpoint invariant representations.
BibTeX
@inproceedings{piergiovanni2021recognizing,
  author = {A. J. Piergiovanni and Michael S. Ryoo},
  title = {{Recognizing Actions in Videos from Unseen Viewpoints}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2021},
  url = {https://openreview.net/forum?id=EYltmRSRyM},
}
Figure