Abstract
As audiences demand greater immersion in film and television, the need for efficient, real-time human motion capture has become paramount. This paper explores the application of deep learning (DL) and Virtual Reality (VR) technology in human-computer interaction (HCI) for media production. We propose a streamlined 2D and 3D human motion recognition system based on Convolutional Neural Networks (CNN). By utilizing an end-to-end architecture that avoids slow intermediate processing, our system achieves a 7–9x speedup over industry-standard methods like OpenPose. While incurring a minor 2.4% drop in accuracy, the system operates at 44.3 ms per frame, offering a practical solution for real-time digital character mapping in professional production environments.
The Challenge of Real-Time Immersion
In modern cinema, "verisimilitude"—the appearance of being true or real—is essential for audience psychological engagement. Achieving this often involves mapping an actor's movements onto a digital avatar. However, traditional motion capture (mocap) is often a bottleneck. Marker-based systems require expensive suits and sensors that restrict movement and complicate the filming process.
Markerless computer vision alternatives, such as OpenPose, offer more freedom but are often too computationally intensive for real-time use. High-accuracy modes can take nearly a second to process a single frame, making them unsuitable for live feedback on set. The core problem addressed here is the need for a "complete and impeccable" scheme that extracts motion data rapidly enough for real-time production without specialized hardware.
A Streamlined Deep Learning Architecture
To address the efficiency gap, we developed a two-stage approach using CNNs.
2D Recognition System
The foundation of the system is a 2D joint detection model using ResNet34 and a Feature Pyramid Network (FPN). Unlike traditional methods that might involve multiple "zooms" or intermediate steps, our model detects human joints at multiple scales in a single execution. It generates "Heatmaps" for joint probability and "Offset maps" for relative distances between joints. This end-to-end design is the primary driver of the system's speed.
3D Recognition and Relative Depth
The 3D system extends the 2D data by estimating depth. A key innovation is the focus on relative depth rather than absolute distance from the camera. Because CNNs process local features, it is more efficient for the network to learn the spatial relationship between connected joints (e.g., the elbow relative to the shoulder) than the absolute position in a 3D scene. A mapping function $F(X, Y, D)$ then synthesizes these coordinates into a final 3D pose.
Results: Prioritizing Speed for Production
The system was validated using the Microsoft COCO dataset for 2D initialization and the Joint Track Auto (JTA) dataset—rendered in the Grand Theft Auto 5 engine—for 3D training.
The results highlight a significant performance leap:
- Computational Efficiency: The 3D motion recognition runs at 44.3 ms, compared to OpenPose’s 138.7 ms (high-speed) and 794.5 ms (high-accuracy).
- Hardware Utility: On a standard GTX 1060, the proposed algorithm achieved a performance score nearly 10 times higher than OpenPose.
- Accuracy Trade-off: The Mean Per Joint Position Error (MPJPE) was 81.2 mm. While OpenPose achieves 68.7 mm, the 2.4% difference in detection accuracy is negligible in a production context where real-time feedback is more valuable than perfect precision.
Conclusion and Future Outlook
By prioritizing speed and "visual psychological reality," this deep-learning framework enables a more seamless transition from physical performance to digital animation. While the current model relies heavily on synthetic data from the JTA dataset, which may not capture all real-world complexities, it provides a robust foundation for markerless mocap.
Future work will focus on improving performance in crowded scenes with significant occlusions and exploring the integration of 6G technology to enable cloud-based rendering, further lowering the barrier to entry for high-quality VR and AI-driven film production.
Summary updated