Recognizing Actions in Videos From Unseen Viewpoints

A. J. Piergiovanni, Michael S. Ryoo

Abstract

Standard methods for video recognition use large CNNs designed to capture spatio-temporal data. However, training these models requires a large amount of labeled training data, containing a wide variety of actions, scenes, settings and camera viewpoints. In this paper, we show that current convolutional neural network models are unable to recognize actions from camera viewpoints not present in their training data (i.e., unseen view action recognition). To address this, we develop approaches based on 3D pose and introduce a new geometric convolutional layer that can learn viewpoint invariant representations. Further, we introduce a new, challenging dataset for unseen view recognition and show the approaches ability to learn viewpoint invariant representations.

Introduction

Current video recognition models are surprisingly brittle: change the camera angle, and their accuracy plummets. While standard Convolutional Neural Networks (CNNs) excel when test data mirrors training data, they struggle with distribution shifts, specifically regarding camera viewpoints. For instance, a model trained exclusively on broadcast footage of a baseball swing (side view) often fails to recognize the same action when captured from a smartphone behind the batter or within a batting cage.

Standard broadcast view used in typical training sets.

Humans naturally construct 3D mental models of actions that remain invariant to perspective. In contrast, CNNs often overfit to specific 2D patterns. Even large-scale datasets like Kinetics lack the viewpoint annotations necessary to force models to generalize to "unseen" angles. This research aims to bridge this gap by introducing a geometric inductive bias into the network architecture.

Methodology

The authors propose the Neural Projection Layer (NPL), which lifts abstract 2D feature maps into a latent 3D world representation. Unlike explicit 3D pose estimation, which can be brittle and ignores crucial objects (e.g., bats or cups), the NPL operates on general features.

The NPL process involves:

  1. Lifting: Estimating a 3D coordinate for every feature point.
  2. Transformation: Learning a rotation ($R$) and translation ($t$) to align the video into a common 3D space.
  3. Projection: Projecting the 3D representation back into multiple 2D "virtual" views.
  4. Optimization: Using a "3D loss" to penalize inconsistent 3D representations of the same action across different videos, alongside regularization to ensure diverse virtual camera placements.

By making these operations differentiable, the model learns an optimal "world" representation via standard gradient descent without requiring ground-truth 3D data or camera parameters.

Datasets and Evaluation

The approach was evaluated on several multi-view datasets, including Human3.6M, Toyota SmartHome (TSH), and NTU-RGB-D. Crucially, the authors introduced an extended MLB-YouTube dataset. They augmented the original broadcast-heavy dataset with 500 "in-the-wild" videos featuring viewpoints such as cell phone footage from stands and batting cages.

An example of an "unseen" viewpoint from a batting cage, which poses a challenge for standard models.

Results

The NPL approach consistently outperformed standard baselines on unseen viewpoints:

  • MLB-YouTube (Unseen Views): Accuracy improved from 27.3% (ResNet-50 baseline) to 42.7% with the NPL.
  • Human3.6M: On unseen views, the baseline performed near random chance (9.1%), while the NPL reached 87.5%.
  • Efficiency: The NPL adds minimal overhead, processing a 3-second clip in 120ms (vs. 105ms for the baseline), making it viable for real-time applications.

A key insight is that standard CNNs do not "accidentally" learn 3D invariance even with massive data; architectural changes are required. Furthermore, the NPL's ability to lift all features—not just human joints—proved vital for actions defined by object interaction, where pose-only methods often failed.

Conclusion

The Neural Projection Layer provides a robust mechanism for learning viewpoint-invariant representations by lifting 2D features into a latent 3D space. This geometric approach significantly improves action recognition in "unseen" viewpoints, moving closer to the perspective-invariant capabilities of human vision. Future work will address challenges like heavy occlusions and highly dynamic camera movements.

Summary updated

Abstract

Standard methods for video recognition use large CNNs designed to capture spatio-temporal data. However, training these models requires a large amount of labeled training data, containing a wide variety of actions, scenes, settings and camera viewpoints. In this paper, we show that current convolutional neural network models are unable to recognize actions from camera viewpoints not present in their training data (i.e., unseen view action recognition). To address this, we develop approaches based on 3D pose and introduce a new geometric convolutional layer that can learn viewpoint invariant representations. Further, we introduce a new, challenging dataset for unseen view recognition and show the approaches ability to learn viewpoint invariant representations.
BibTeX
@inproceedings{piergiovanni2021recognizing,
  author = {A. J. Piergiovanni and Michael S. Ryoo},
  title = {{Recognizing Actions in Videos From Unseen Viewpoints}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2021},
  url = {https://openreview.net/forum?id=EYltmRSRyM},
}
Figure