Exploring the Impact of Rendering Method and Motion Quality on Model Performance when Using Multi-view Synthetic Data for Action Recognition

Stanislav Panev, Emily Kim, Sai Abhishek Si Namburu, Desislava Nikolova, Celso de Melo, Fernando De la Torre, Jessica K. HodginsWACV 2024 · DOI

Abstract

Human Action Recognition (HAR) is essential for applications ranging from surveillance to human-computer interaction. However, training robust models requires massive, multi-view labeled datasets that are logistically difficult to collect. This paper introduces REMAG (REndering, Motion, Aerial, and Ground), a dataset suite designed to systematically evaluate the impact of rendering quality and motion fidelity on the "domain gap" between synthetic and real data. Our findings reveal that high-quality motion capture (MoCap) data is more critical for model performance than photorealistic neural rendering. Furthermore, we demonstrate that synthetic pre-training significantly reduces the need for large-scale real-world data, with models fine-tuned on minimal real samples outperforming those trained solely on real data.

The Challenge of Real-World HAR

Developing effective HAR models involves teaching computers to interpret complex human movements, such as waving, walking, or carrying specific objects. Real-world data collection is fraught with challenges: it requires diverse actors, multiple synchronized camera angles (including dynamic aerial views), and labor-intensive manual labeling. Synthetic data generated via computer graphics (CG) offers a scalable solution, but its effectiveness depends on how well it bridges the "domain gap"—the difference in appearance and movement between simulated and real humans.

The REMAG Dataset Suite

To address these challenges, the researchers developed the REMAG dataset. It features 11 activity classes, including both simple gestures and complex object-handling tasks. The data is captured from three synchronized viewpoints: two static ground-level cameras and one orbiting Unmanned Aerial Vehicle (UAV).

The REMAG dataset suite provides synchronized views across real and synthetic domains, including standard CG and neural rendering (GAN-based) approaches.

Methodology: Isolating Rendering and Motion

The core of the REMAG study is a 2x2 experimental matrix designed to isolate the factors contributing to the domain gap:

  1. Rendering Methods:
    • Standard CG: Utilizes the Blender Eevee engine with 3D rigged characters.
  2. Neural Rendering: Employs Liquid Warping GAN (LWG) to synthesize motion onto real human images, aiming for higher photorealism.
  3. Motion Sources:
    • Motion Capture (MC): High-fidelity data obtained from professional MoCap suits.
  4. Video-extracted (RGB): Motion estimated from 2D video using the VIBE algorithm, which is more accessible but contains more noise.

The researchers evaluated these variants using state-of-the-art models: SlowFast, MViT, and X3D.

Key Insights

Motion Quality vs. Rendering Quality

A surprising result of the study is that Motion Quality > Rendering Quality. Despite the visual sophistication of neural rendering, standard CG rendering yielded better model performance. The authors observed that the temporal artifacts and "jitter" introduced by the GAN-based synthesis were more detrimental to the models' ability to learn actions than the "cleaner," albeit less photorealistic, look of standard CGI.

The Aerial Advantage

Data captured from the orbiting UAV proved to be a significant asset. Because the drone circles the subject, the resulting data encompasses a continuous range of angles. Models trained on this aerial data were found to be far more robust, transferring their knowledge to ground-level views much more effectively than ground-trained models transferred to aerial views.

Confounding Pairs and Fine-Grained Recognition

The dataset intentionally includes "confounding pairs"—activities that look similar, such as carrying a shovel versus carrying a baseball bat. This ensures that models are learning specific motion patterns and object interactions rather than relying on simple visual cues.

An example of a fine-grained action in the REMAG dataset: carrying a shovel. Such activities test the model's ability to distinguish between similar object-handling motions.

Results and Transfer Learning

The study highlights the power of synthetic pre-training. Models pre-trained on REMAG's synthetic data and fine-tuned on just 10 real subjects achieved higher accuracy than models trained from scratch on the entire real-world dataset. Overall, models trained purely on synthetic data reached accuracies within 4–8% of those trained on real data, establishing synthetic data as a highly viable substitute.

Conclusion

The REMAG suite demonstrates that high-fidelity motion and diverse camera perspectives are the most critical components for effective synthetic HAR data. By prioritizing clean geometry and MoCap-driven motion, researchers can significantly lower the barriers to training robust action recognition models. Future research will focus on increasing activity diversity and exploring advanced generative techniques to further narrow the domain gap.

Summary updated

BibTeX
@inproceedings{panev2024exploring,
  author = {Stanislav Panev and Emily Kim and Sai Abhishek Si Namburu and Desislava Nikolova and Celso de Melo and Fernando De la Torre and Jessica K. Hodgins},
  title = {{Exploring the Impact of Rendering Method and Motion Quality on Model Performance when Using Multi-view Synthetic Data for Action Recognition}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
  doi = {10.1109/wacv57701.2024.00453},
}
Figure