The Use of Deep Learning and VR Technology in Film and Television Production From the Perspective of Audience Psychology

Yangfan Tong, Weiran Cao, Qian Sun, Dong ChenFrontiers in Psychology 2021 · DOI

Abstract

As audiences demand greater immersion in film and television, the need for efficient, real-time human motion capture has become paramount. This paper explores the application of deep learning (DL) and Virtual Reality (VR) technology in human-computer interaction (HCI) for media production. We propose a streamlined 2D and 3D human motion recognition system based on Convolutional Neural Networks (CNN). By utilizing an end-to-end architecture that avoids slow intermediate processing, our system achieves a 7–9x speedup over industry-standard methods like OpenPose. While incurring a minor 2.4% drop in accuracy, the system operates at 44.3 ms per frame, offering a practical solution for real-time digital character mapping in professional production environments.

The Challenge of Real-Time Immersion

In modern cinema, "verisimilitude"—the appearance of being true or real—is essential for audience psychological engagement. Achieving this often involves mapping an actor's movements onto a digital avatar. However, traditional motion capture (mocap) is often a bottleneck. Marker-based systems require expensive suits and sensors that restrict movement and complicate the filming process.

Markerless computer vision alternatives, such as OpenPose, offer more freedom but are often too computationally intensive for real-time use. High-accuracy modes can take nearly a second to process a single frame, making them unsuitable for live feedback on set. The core problem addressed here is the need for a "complete and impeccable" scheme that extracts motion data rapidly enough for real-time production without specialized hardware.

A Streamlined Deep Learning Architecture

To address the efficiency gap, we developed a two-stage approach using CNNs.

2D Recognition System

The foundation of the system is a 2D joint detection model using ResNet34 and a Feature Pyramid Network (FPN). Unlike traditional methods that might involve multiple "zooms" or intermediate steps, our model detects human joints at multiple scales in a single execution. It generates "Heatmaps" for joint probability and "Offset maps" for relative distances between joints. This end-to-end design is the primary driver of the system's speed.

Figure 1: Comparison between the proposed end-to-end scheme and traditional multi-stage methods.

3D Recognition and Relative Depth

The 3D system extends the 2D data by estimating depth. A key innovation is the focus on relative depth rather than absolute distance from the camera. Because CNNs process local features, it is more efficient for the network to learn the spatial relationship between connected joints (e.g., the elbow relative to the shoulder) than the absolute position in a 3D scene. A mapping function $F(X, Y, D)$ then synthesizes these coordinates into a final 3D pose.

Results: Prioritizing Speed for Production

The system was validated using the Microsoft COCO dataset for 2D initialization and the Joint Track Auto (JTA) dataset—rendered in the Grand Theft Auto 5 engine—for 3D training.

The results highlight a significant performance leap:

  • Computational Efficiency: The 3D motion recognition runs at 44.3 ms, compared to OpenPose’s 138.7 ms (high-speed) and 794.5 ms (high-accuracy).
  • Hardware Utility: On a standard GTX 1060, the proposed algorithm achieved a performance score nearly 10 times higher than OpenPose.
  • Accuracy Trade-off: The Mean Per Joint Position Error (MPJPE) was 81.2 mm. While OpenPose achieves 68.7 mm, the 2.4% difference in detection accuracy is negligible in a production context where real-time feedback is more valuable than perfect precision.
Figure 6: 3D pose estimation and joint modeling applied to a human subject.

Conclusion and Future Outlook

By prioritizing speed and "visual psychological reality," this deep-learning framework enables a more seamless transition from physical performance to digital animation. While the current model relies heavily on synthetic data from the JTA dataset, which may not capture all real-world complexities, it provides a robust foundation for markerless mocap.

Future work will focus on improving performance in crowded scenes with significant occlusions and exploring the integration of 6G technology to enable cloud-based rendering, further lowering the barrier to entry for high-quality VR and AI-driven film production.

Summary updated

Abstract

As the development of artificial intelligence (AI) technology, the deep-learning (DL)-based Virtual Reality (VR) technology, and DL technology are applied in human-computer interaction (HCI), and their impacts on modern film and TV works production and audience psychology are analyzed. In film and TV production, audiences have a higher demand for the verisimilitude and immersion of the works, especially in film production. Based on this, a 2D image recognition system for human body motions and a 3D recognition system for human body motions based on the convolutional neural network (CNN) algorithm of DL are proposed, and an analysis framework is established. The proposed systems are simulated on practical and professional datasets, respectively. The results show that the algorithm's computing performance in 2D image recognition is 7-9 times higher than that of the Open Pose method. It runs at 44.3 ms in 3D motion recognition, significantly lower than the Open Pose method's 794.5 and 138.7 ms. Although the detection accuracy has dropped by 2.4%, it is more efficient and convenient without limitations of scenarios in practical applications. The AI-based VR and DL enriches and expands the role and application of computer graphics in film and TV production using HCI technology theoretically and practically.
BibTeX
@inproceedings{tong2021use,
  author = {Yangfan Tong and Weiran Cao and Qian Sun and Dong Chen},
  title = {{The Use of Deep Learning and VR Technology in Film and Television Production From the Perspective of Audience Psychology}},
  booktitle = {Frontiers in Psychology},
  year = {2021},
  doi = {10.3389/fpsyg.2021.634993},
}
Figure