Path-Analysis-Based Reinforcement Learning Algorithm for Imitation Filming

Yuanjie Dang, Chong Huang, Peng Chen, Ronghua Liang, Xin Yang, Kwang‐Ting Cheng

Abstract

Imitation filming has been applied to autonomous filming by mimicking human operators. To imitate the operation of cameramen when filming multiple human actions, existing methods plan the camera motion through time series prediction or train multiple models to handle a particular style in a specific situation. As a result, these methods require various settings to adapt to different scenarios. In this work, we overcome such limitations and propose an end-to-end imitation learning framework for drone cinematography systems. The framework consists of two main components: ( <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</b> ) an efficient motion feature extraction module for generating a compact motion feature space, ( <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</b> ) a path-analysis-based reinforcement learning (PABRL) algorithm for imitating multiple filming styles from demonstrations and incorporating aesthetical features for improved perspective shots. Our PABRL method is based on the actor–critic network, which regards multiple human motion variables, camera translations, and image composition as inputs and then outputs an aesthetical filming strategy related to the subject motion. In addition, we propose an attention mechanism and a long–short-term rewarding function to enhance the motion feature space and the integrity of the generated trajectory, respectively. Extensive experimental results in simulated and real outdoor environments demonstrate that compared with state-of-the-art methods, our method can achieve 69.8% higher performance in terms of trajectory planning accuracy while successfully incorporating aesthetical features into the captured videos.
BibTeX
@article{dang2023path,
  author = {Yuanjie Dang and Chong Huang and Peng Chen and Ronghua Liang and Xin Yang and Kwang‐Ting Cheng},
  title = {{Path-Analysis-Based Reinforcement Learning Algorithm for Imitation Filming}},
  journal = {IEEE Trans. Multimedia},
  year = {2023},
  doi = {10.1109/tmm.2022.3151463},
}

Loading related content...

Figure