Learning to Film From Professional Human Motion Videos.

Chong Huang, Chuan-En Lin, Zhenyu Yang, Yan Kong, Peng Chen, Xin Yang, Kwang-Ting ChengCVPR 2019 · OpenReview

Abstract: We investigate the problem of 6 degrees of freedom (DOF) camera planning for filming professional human motion videos using a camera drone. Existing methods either plan motions for only a pan-tilt-zoom (PTZ) camera, or adopt ad-hoc solutions without carefully considering the impact of video contents and previous camera motions on the future camera motions. In this study, we propose a learning-based framework which incorporates the video contents and previous camera motions to predict the future camera motions that enable the capture of professional videos. Specifically, the inputs of our framework are video contents which are represented using subject-related feature based on 2D skeleton and scene-related features extracted from background RGB images, and camera motions which are represented using optical flows. The correlation between the inputs and output future camera motions are learned via a sequence-to-sequence convolutional long short-term memory (Seq2Seq ConvLSTM) network from a large set of video clips. We deploy our approach to a real drone cinematography system by first predicting the future camera motions, and then converting them to the drone's control commands via an odometer. Our experimental results on extensive datasets and showcases exhibit significant improvements in our approach over conventional baselines and our approach can successfully mimic the footage of a professional cameraman.

Introduction

Autonomous drone cinematography has traditionally relied on hard-coded rules or simple "follow-me" logic. While functional, these methods often lack the aesthetic nuance of a professional filmmaker. This research introduces a novel approach that learns the "art" of filming by observing professional human motion videos, treating cinematography as an imitation learning problem.

The Challenge of 6DOF Planning

Capturing high-quality video of a moving person is complex. Current consumer solutions, such as DJI’s QuickShot, typically use "parametric" methods—predefined paths like circles or spirals. These methods are often "blind" to artistic context; they do not adjust for environmental factors like terrain or the subject's specific movements, leading to repetitive footage.

Professional cinematography requires dynamic 6-degree-of-freedom (6DOF) camera planning. A human operator adjusts position and rotation based on the scene's context and the subject's intent. Training an autonomous system for this is difficult because professional footage rarely includes the raw drone telemetry (GPS/IMU data) necessary for direct supervised learning.

Proposed Framework

The researchers developed a learning-based framework that treats cinematography as a sequence-to-sequence prediction problem.

The proposed learning-based framework for drone cinematography, illustrating the pipeline from feature extraction to motion estimation.

Feature Extraction

The system processes three distinct input streams:

  • Subject Motion: Extracts a 13-keypoint 2D skeleton using OpenPose, convolved with a Gaussian kernel to create "motion maps."
  • Background Scene: A 3-layer CNN extracts spatial features from RGB frames to provide environmental context.
  • Previous Camera Motion: Represented via dense optical flow, capturing relative movement.

Prediction and Estimation

These features are fed into a Seq2Seq ConvLSTM network. The encoder analyzes a window of past frames, while the decoder predicts future camera motion as optical flow maps. To translate these pixel-level predictions into drone commands, the authors use Essential Matrix decomposition to estimate 3D rotations and translations. Finally, a "min-snap" polynomial algorithm ensures smooth, feasible physical trajectories.

Learning from Professionals

To train the model, the authors curated a new dataset of 92 high-quality outdoor video clips from Getty Images. These clips were manually categorized into six distinct professional filming styles, such as circling or side-following.

Six distinct professional filming styles (A-F) identified in the dataset, used to train the model in diverse cinematic techniques.

Through data augmentation techniques like sliding windows and horizontal flipping, the training pool was expanded to over 11,000 clips, allowing the model to learn a wide variety of cinematic responses to human motion.

Key Insights and Results

  • Multimodal Fusion: The study found that integrating background and skeletal pose data significantly reduced prediction errors compared to using optical flow alone. The background informs the drone of navigable space, while the skeleton helps anticipate the subject's next move.
  • Temporal Context: Optimal prediction was achieved using a 9-frame history (at 3fps), aligning with psychological theories on human temporal perception.
  • Performance: In a user study with 10 volunteers, the proposed system received a cinematic quality score of 4.29/5.0, significantly outperforming DJI’s ActiveTrack (2.64) and previous state-of-the-art research (3.15).

Conclusion and Future Work

The framework was successfully deployed on a DJI Matrice 100 drone equipped with an Nvidia Jetson TX2, demonstrating real-time performance on edge hardware. While the system shows great promise in mimicking professional cameramen, future work will address current limitations such as scale ambiguity, multi-subject tracking, and handling prolonged occlusions.

Summary updated

Abstract

We investigate the problem of 6 degrees of freedom (DOF) camera planning for filming professional human motion videos using a camera drone. Existing methods either plan motions for only a pan-tilt-zoom (PTZ) camera, or adopt ad-hoc solutions without carefully considering the impact of video contents and previous camera motions on the future camera motions. As a result, they can hardly achieve satisfactory results in our drone cinematography task. In this study, we propose a learning-based framework which incorporates the video contents and previous camera motions to predict the future camera motions that enable the capture of professional videos. Specifically, the inputs of our framework are video contents which are represented using subject-related feature based on 2D skeleton and scene-related features extracted from background RGB images, and camera motions which are represented using optical flows. The correlation between the inputs and output future camera motions are learned via a sequence-to-sequence convolutional long short-term memory (Seq2Seq ConvLSTM) network from a large set of video clips. We deploy our approach to a real drone cinematography system by first predicting the future camera motions, and then converting them to the drone's control commands via an odometer. Our experimental results on extensive datasets and showcases exhibit significant improvements in our approach over conventional baselines and our approach can successfully mimic the footage of a professional cameraman.
BibTeX
@inproceedings{huang2019learning,
  author = {Chong Huang and Chuan-En Lin and Zhenyu Yang and Yan Kong and Peng Chen and Xin Yang and Kwang-Ting Cheng},
  title = {{Learning to Film From Professional Human Motion Videos.}},
  booktitle = {CVPR},
  year = {2019},
  url = {https://openreview.net/forum?id=rQNNSyQeupH},
}

Loading related content...

Figure