Synthetic Humans for Action Recognition from Unseen Viewpoints

Gül Varol, Ivan Laptev, Cordelia Schmid, Andrew Zisserman

Abstract

Although synthetic training data has been shown to be beneficial for tasks such as human pose estimation, its use for RGB human action recognition is relatively unexplored. Our goal in this work is to answer the question whether synthetic humans can improve the performance of human action recognition, with a particular focus on generalization to unseen viewpoints. We make use of the recent advances in monocular 3D human body reconstruction from real action sequences to automatically render synthetic training videos for the action labels. We investigate the extent of variations and augmentations that are beneficial to improving performance at new viewpoints, including changes in body shape, clothing, and motion-relevant augmentations. We introduce a new data generation methodology, SURREACT, that allows training of spatio-temporal CNNs for action classification. We substantially improve the state-of-the-art action recognition performance on the NTU RGB+D and UESTC standard human action multi-view benchmarks and demonstrate improvements in one-shot training scenarios on Kinetics.

Introduction: The Viewpoint Generalization Problem

Standard action recognition models are notoriously sensitive to camera angles. While massive datasets of people performing actions exist, they are rarely captured from every possible angle. Collecting "all-view" data for every human activity is a logistical challenge due to the combinatorial explosion of variables like clothing, body shape, and lighting. This leads to the viewpoint generalization problem, where a system trained on front-facing cameras fails when the viewpoint shifts to the side. This research investigates whether 3D computer graphics can bridge these gaps by teaching models to recognize actions regardless of camera placement.

Methodology: The SURREACT Pipeline

The authors introduce a pipeline to create the SURREACT (Synthetic hUmans foR REal ACTions) dataset. The process leverages monocular 3D reconstruction to extract motion from real videos and then augments this data to train robust spatio-temporal CNNs.

Overview of the SURREACT pipeline for generating synthetic training data from real-world motion.

The pipeline follows four main steps:

  1. 3D Motion Extraction: Monocular 3D reconstruction tools (e.g., HMMR, VIBE) extract 3D body movements (SMPL parameters) directly from existing real-world videos.
  2. Augmentation: Diversity is increased by adding "video-level noise" to joint rotations and interpolating between different individuals performing the same action to create new, plausible movement patterns.
  3. Rendering: These 3D models are rendered into synthetic videos with randomized clothing, body shapes, backgrounds, and eight different camera viewpoints distributed 360° around the performer.
  4. Non-Uniform Sampling: During training, frames are randomly sampled across the entire video while maintaining temporal order. This makes the model robust to variations in action speed and helps capture long-term context.

Experimental Setup and Benchmarks

The approach was evaluated on several major benchmarks:

  • NTU RGB+D: A large-scale dataset with 60 action classes. The study focused on a "Cross-View-Subject" protocol to test generalization to a 90° side view.
  • UESTC: A dataset designed for multi-view analysis, covering 40 actions from 8 viewpoints.
  • Kinetics-15: A subset of the "in-the-wild" Kinetics dataset used to test "one-shot" scenarios where only a single real training video is available per class.

Analysis and Results

The study yielded several key insights:

  • Pixels over Parameters: Training on rendered images of synthetic humans outperformed training on raw 3D joint coordinates. Image-based models better leverage standard data augmentations and bridge the domain gap when mixed with real video data.
  • Motion Diversity: Varying the motion (interpolation and noise) provided a more significant performance boost than varying appearance (clothing or body shape).
  • Non-Uniform Sampling: This simple strategy made 3D CNNs speed-invariant and significantly improved results compared to standard consecutive frame sampling.

Performance Gains

Synthetic data proved to be a powerful supplement to real data:

  • NTU RGB+D: On the challenging 90° unseen viewpoint, adding synthetic data boosted RGB accuracy from 53.6% to 69.0%.
  • UESTC: Average performance across all views increased from 49.4% to 66.4%.
  • One-Shot Learning: In the Kinetics-15 experiment, adding synthetic augmentations improved accuracy by approximately 6.5%.

Conclusion and Future Work

While synthetic data significantly improves viewpoint generalization, challenges remain. The current pipeline focuses on "body-only" movements and struggles with actions involving objects (e.g., "drinking"). Furthermore, the quality of synthetic data is dependent on the accuracy of the initial 3D pose estimation. Future work may involve generative models that simulate entire scene contexts and object interactions to further enhance the realism and utility of synthetic training data.

Summary updated

Abstract

Although synthetic training data has been shown to be beneficial for tasks such as human pose estimation, its use for RGB human action recognition is relatively unexplored. Our goal in this work is to answer the question whether synthetic humans can improve the performance of human action recognition, with a particular focus on generalization to unseen viewpoints. We make use of the recent advances in monocular 3D human body reconstruction from real action sequences to automatically render synthetic training videos for the action labels. We make the following contributions: (i) we investigate the extent of variations and augmentations that are beneficial to improving performance at new viewpoints. We consider changes in body shape and clothing for individuals, as well as more action relevant augmentations such as non-uniform frame sampling, and interpolating between the motion of individuals performing the same action; (ii) We introduce a new data generation methodology, SURREACT, that allows training of spatio-temporal CNNs for action classification; (iii) We substantially improve the state-of-the-art action recognition performance on the NTU RGB+D and UESTC standard human action multi-view benchmarks; Finally, (iv) we extend the augmentation approach to in-the-wild videos from a subset of the Kinetics dataset to investigate the case when only one-shot training data is available, and demonstrate improvements in this case as well.
BibTeX
@article{varol2021synthetic,
  author = {Gül Varol and Ivan Laptev and Cordelia Schmid and Andrew Zisserman},
  title = {{Synthetic Humans for Action Recognition from Unseen Viewpoints}},
  journal = {Int. J. Comput. Vis.},
  year = {2021},
  url = {https://openreview.net/forum?id=UnZ2cBAB-bq},
}
Figure