Abstract
Although synthetic training data has been shown to be beneficial for tasks such as human pose estimation, its use for RGB human action recognition is relatively unexplored. Our goal in this work is to answer the question whether synthetic humans can improve the performance of human action recognition, with a particular focus on generalization to unseen viewpoints. We make use of the recent advances in monocular 3D human body reconstruction from real action sequences to automatically render synthetic training videos for the action labels. We investigate the extent of variations and augmentations that are beneficial to improving performance at new viewpoints, including changes in body shape, clothing, and motion-relevant augmentations. We introduce a new data generation methodology, SURREACT, that allows training of spatio-temporal CNNs for action classification. We substantially improve the state-of-the-art action recognition performance on the NTU RGB+D and UESTC standard human action multi-view benchmarks and demonstrate improvements in one-shot training scenarios on Kinetics.
Introduction: The Viewpoint Generalization Problem
Standard action recognition models are notoriously sensitive to camera angles. While massive datasets of people performing actions exist, they are rarely captured from every possible angle. Collecting "all-view" data for every human activity is a logistical challenge due to the combinatorial explosion of variables like clothing, body shape, and lighting. This leads to the viewpoint generalization problem, where a system trained on front-facing cameras fails when the viewpoint shifts to the side. This research investigates whether 3D computer graphics can bridge these gaps by teaching models to recognize actions regardless of camera placement.
Methodology: The SURREACT Pipeline
The authors introduce a pipeline to create the SURREACT (Synthetic hUmans foR REal ACTions) dataset. The process leverages monocular 3D reconstruction to extract motion from real videos and then augments this data to train robust spatio-temporal CNNs.
The pipeline follows four main steps:
- 3D Motion Extraction: Monocular 3D reconstruction tools (e.g., HMMR, VIBE) extract 3D body movements (SMPL parameters) directly from existing real-world videos.
- Augmentation: Diversity is increased by adding "video-level noise" to joint rotations and interpolating between different individuals performing the same action to create new, plausible movement patterns.
- Rendering: These 3D models are rendered into synthetic videos with randomized clothing, body shapes, backgrounds, and eight different camera viewpoints distributed 360° around the performer.
- Non-Uniform Sampling: During training, frames are randomly sampled across the entire video while maintaining temporal order. This makes the model robust to variations in action speed and helps capture long-term context.
Experimental Setup and Benchmarks
The approach was evaluated on several major benchmarks:
- NTU RGB+D: A large-scale dataset with 60 action classes. The study focused on a "Cross-View-Subject" protocol to test generalization to a 90° side view.
- UESTC: A dataset designed for multi-view analysis, covering 40 actions from 8 viewpoints.
- Kinetics-15: A subset of the "in-the-wild" Kinetics dataset used to test "one-shot" scenarios where only a single real training video is available per class.
Analysis and Results
The study yielded several key insights:
- Pixels over Parameters: Training on rendered images of synthetic humans outperformed training on raw 3D joint coordinates. Image-based models better leverage standard data augmentations and bridge the domain gap when mixed with real video data.
- Motion Diversity: Varying the motion (interpolation and noise) provided a more significant performance boost than varying appearance (clothing or body shape).
- Non-Uniform Sampling: This simple strategy made 3D CNNs speed-invariant and significantly improved results compared to standard consecutive frame sampling.
Performance Gains
Synthetic data proved to be a powerful supplement to real data:
- NTU RGB+D: On the challenging 90° unseen viewpoint, adding synthetic data boosted RGB accuracy from 53.6% to 69.0%.
- UESTC: Average performance across all views increased from 49.4% to 66.4%.
- One-Shot Learning: In the Kinetics-15 experiment, adding synthetic augmentations improved accuracy by approximately 6.5%.
Conclusion and Future Work
While synthetic data significantly improves viewpoint generalization, challenges remain. The current pipeline focuses on "body-only" movements and struggles with actions involving objects (e.g., "drinking"). Furthermore, the quality of synthetic data is dependent on the accuracy of the initial 3D pose estimation. Future work may involve generative models that simulate entire scene contexts and object interactions to further enhance the realism and utility of synthetic training data.
Summary updated