A simulation environment for drone cinematography

Fan Zhang, David Hall, Tao Xu, Stephen Boyle, David R. BullarXiv 2020 · OpenReview

TL;DR

This paper introduces a novel workflow for creating realistic drone cinematography simulations using Unreal Engine 4. By combining environmental image capture, 3D reconstruction via photogrammetry, and a user-friendly interface, the tool enables detailed shot planning, rehearsal, and training for complex drone operations, especially for live events.

What Problem Does This Paper Address?

Planning and rehearsing drone cinematography, particularly for live events where opportunities are limited, presents a significant challenge. Existing simulation tools often lack realistic environments, robust shot planning features, or the ability to integrate dynamic foreground elements. This gap hinders effective pre-visualization, operator training, and ultimately, the quality of the final footage. The authors aim to bridge this by developing a simulation environment that leverages real-world data for highly realistic backgrounds and offers flexible tools for planning and rehearsal.

Methods

The core of the paper presents a three-step workflow:

  1. Environmental Image Capture: This involves determining optimal scanning strategies and flight parameters for capturing imagery. Sources can be virtual globe software or actual aerial photography. The paper details considerations for flight trajectory (grid scanning), flight heights (multi-layer approach for comprehensive coverage), and viewing angles, highlighting intermediate angles as providing the best balance for reconstruction.
  2. 3D Model Reconstruction: Captured images are processed using photogrammetry software (specifically Pix4D Mapper was chosen for its efficiency and quality) to generate 3D background environment models. The paper also discusses pre-processing (manual outlier rejection) and post-processing (using Blender for smoothing, flattening, and texture painting) to refine these models and address artefacts.
  3. User-Friendly Simulation Interface: A standalone application built in Unreal Engine 4 provides three modes: editing, simulation, and free play. This interface allows users to import reconstructed environments, add and manipulate foreground objects (drones, cars, cyclists, boats), define programmable scenarios, and plan/rehearse various shot types (e.g., ESTABLISH, CHASE, FLYBY).

Datasets

The paper does not explicitly list or introduce new benchmark datasets. Instead, it utilizes:

  • Virtual Globe Software: Such as Google Earth and Microsoft Bing Maps for initial environmental data.
  • Real-world Aerial Photography: Captured using drones or helicopters.
  • Pre-generated 3D Environments: Specifically, "Clifton Downs" and "Harbourside" in Bristol, UK, reconstructed from Google Earth data.
  • Foreground Object Models: Including drones, cars, cyclists, and boats.
  • UE4 Marketplace Assets: Specifically, the "Country Side" environment asset was used for testing reconstruction parameters.

Key Insights and Ideas

  • Multi-layer height scanning is crucial: For comprehensive 3D reconstruction, especially in urban or complex landscapes, scanning at multiple altitudes (base, average building height, maximum building height) significantly improves coverage and detail capture.
  • Intermediate viewing angles yield better reconstructions: While high angles capture local details and low angles capture landscape shape, intermediate angles (e.g., 85/60/35 degrees) offer the best overall balance for reconstructing both landscape and details with fewer distortions.
  • Overlap ratios are critical for quality: Higher in-track overlap ratios (above 70% for landscape, 80% for details) are necessary to achieve high-quality 3D reconstructions, as indicated by subjective quality scores.
  • Post-processing in Blender is essential: Photogrammetry often leaves artefacts. Tools like smoothing, flattening, and texture painting in Blender are vital for refining the reconstructed 3D models to a usable state for simulation.
  • Unreal Engine 4 is a powerful platform for this application: Its capabilities in rendering, asset integration, and interactive development make it well-suited for creating a realistic and functional simulation environment.

Key Results

  • A functional simulation environment was developed, capable of creating realistic 3D environments from real-world data.
  • The workflow demonstrates effective methods for environmental image capture, 3D reconstruction, and integration into a simulation.
  • The simulation interface supports flexible shot planning, rehearsal, and training with programmable scenarios and dynamic objects.
  • User feedback from a workshop was positive, highlighting the tool's utility for drone cinematography planning and training.

Limitations and Future Work

Limitations:

  • Environment Creation Complexity: The authors acknowledge that streamlining the environment creation process for users to easily create their own environments remains a challenge.
  • Manual Outlier Rejection: For demonstration purposes, simple manual outlier rejection was used instead of more complex in-painting algorithms, which can be time-consuming for large datasets.

Future Work:

  • VR Integration: Integrating the software with VR devices to achieve more immersive visual experiences.
  • Enhanced Environment Creation: Developing a more flexible workflow for users to create 3D maps for specific areas.
  • HMD Interface: Creating an interface for Head-Mounted Displays (HMDs) to enhance immersion.
  • Full Drone Controller Interface: Expanding the free-play mode to support a full drone controller interface.
  • Autonomous Drone Operation Integration: Enabling compatibility with primary auto-pilot software for autonomous drone operation.

Summary updated

Abstract

In this paper, we present a workflow for the simulation of drone operations exploiting realistic background environments constructed within Unreal Engine 4 (UE4). Methods for environmental image capture, 3D reconstruction (photogrammetry) and the creation of foreground assets are presented along with a flexible and user-friendly simulation interface. Given the geographical location of the selected area and the camera parameters employed, the scanning strategy and its associated flight parameters are first determined for image capture. Source imagery can be extracted from virtual globe software or obtained through aerial photography of the scene (e.g. using drones). The latter case is clearly more time consuming but can provide enhanced detail, particularly where coverage of virtual globe software is limited. The captured images are then used to generate 3D background environment models employing photogrammetry software. The reconstructed 3D models are then imported into the simulation interface as background environment assets together with appropriate foreground object models as a basis for shot planning and rehearsal. The tool supports both free-flight and parameterisable standard shot types along with programmable scenarios associated with foreground assets and event dynamics. It also supports the exporting of flight plans. Camera shots can also be designed to provide suitable coverage of any landmarks which need to appear in-shot. This simulation tool will contribute to enhanced productivity, improved safety (awareness and mitigations for crowds and buildings), improved confidence of operators and directors and ultimately enhanced quality of viewer experience.
BibTeX
@misc{zhang2020simulation,
  author = {Fan Zhang and David Hall and Tao Xu and Stephen Boyle and David R. Bull},
  title = {{A simulation environment for drone cinematography}},
  year = {2020},
  eprint = {2010.01315},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2010.01315},
}
Figure