CinemAirSim: A Camera-Realistic Robotics Simulator for Cinematographic Purposes

Pablo Pueyo, Eric Cristofalo, Eduardo Montijano, Mac Schwager

TL;DR

This paper introduces CinemAirSim, a modification of the AirSim robotics simulator that adds realistic cinematic camera capabilities. By integrating a thin-lens camera model and exposing control over parameters like focal length, aperture, and focus distance, CinemAirSim allows for the simulation of artistic camera effects crucial for film and entertainment, opening new avenues for robotics research in cinematography.

What Problem Does This Paper Address?

The increasing use of Unmanned Aerial Vehicles (UAVs) in film and entertainment has highlighted a gap in current robotics simulators. While existing simulators excel at realistic physics and sensor modeling, they typically employ a basic pinhole camera model. This model lacks the ability to simulate artistic camera effects like depth of field and bokeh, which are fundamental to cinematography. This limitation hinders research into developing control and planning algorithms for drones that can produce aesthetically pleasing aerial shots, as it's difficult to test and validate these artistic aspects in a simulated environment without expensive real-world equipment.

Methods

The core contribution is the modification of the AirSim simulator, built on Unreal Engine, to incorporate a cinematic camera model. This was achieved by:

  • Replacing the default pinhole camera: The standard CameraComponent in AirSim was replaced with Unreal Engine's CineCameraComponent.
  • Extending the API: The AirSim API was augmented with new methods to allow real-time control of cinematic camera parameters. This includes:
  • Filmback: Control over the sensor size (width and height) and aspect ratio.
  • Commercial Lenses: Simulation of various lenses with adjustable focal length, aperture, and minimum focus distance.
  • Focus Distance: Real-time adjustment of the plane of focus.
  • Focus Aperture: Control over the f-stop, influencing depth of field and bokeh.
  • Focal Length: Adjustment of zoom and field of view.
  • Leveraging Unreal Engine's capabilities: The simulator utilizes Unreal Engine's rendering and physics engines to provide photo-realistic environments and accurate simulation of camera effects.

Datasets

Not discussed in the paper. The paper focuses on reproducing scenes from award-winning films for qualitative evaluation rather than using specific datasets for training or quantitative benchmarking.

Key Insights and Ideas

  • Bridging the gap between robotics and cinematography: The paper demonstrates that advanced cinematic camera simulation can be integrated into existing robotics simulators, making it accessible for research without the need for expensive real-world equipment.
  • Enabling artistic control in robotics: By providing real-time control over cinematic camera parameters, CinemAirSim allows researchers to explore how artistic camera choices can be incorporated into drone control and planning algorithms.
  • Potential for novel research: The simulator opens doors for research in areas like autonomous cinematography, multi-drone coordinated shots, and the development of control strategies that balance technical performance with aesthetic goals.

Key Results

The paper presents qualitative results demonstrating the simulator's ability to reproduce realistic cinematic effects. Examples include:

  • Simulating depth of field: The simulator can accurately render scenes with selective focus, mimicking the look of real-world cameras.
  • Reproducing movie scenes: The authors successfully recreated scenes from films like "The Revenant" and "Road to Perdition," showcasing the simulator's fidelity in capturing specific photographic styles.
  • Demonstrating real-time control: Experiments show live adjustments of focus distance and focal length, highlighting the responsiveness of the extended API.
  • Qualitative comparison: Comparisons with the standard AirSim camera show a significant improvement in realism and artistic control.

Limitations and Future Work

Limitations:

  • Ground Truth: The authors acknowledge that obtaining precise ground truth camera configurations for real movie scenes is challenging, leading to open-loop trajectory generation for experiments.
  • Focus on Qualitative Results: The primary evaluation is qualitative, demonstrating the visual fidelity and artistic capabilities rather than providing quantitative performance metrics.

Future Work:

  • Autonomous Control Algorithms: Developing control algorithms that can autonomously decide on optimal photographic parameters alongside drone trajectories.
  • Multi-Drone Coordinated Shots: Exploring scenarios where multiple drones coordinate their shots, potentially focusing on different parts of a scene or avoiding filming each other.
  • Further Integration: Expanding the simulator's capabilities to include more advanced camera models and cinematic features.

Summary updated

Abstract

Drones and Unmanned Aerial Vehicles (UAV's) are becoming increasingly popular in the film and entertainment industries in part because of their maneuverability and the dynamic shots and perspectives they enable. While there exists methods for controlling the position and orientation of the drones for visibility, other artistic elements of the filming process, such as focal blur and light control, remain unexplored in the robotics community. The lack of cinemetographic robotics solutions is partly due to the cost associated with the cameras and devices used in the filming industry, but also because state-of-the-art photo-realistic robotics simulators only utilize a full in-focus pinhole camera model which does incorporate these desired artistic attributes. To overcome this, the main contribution of this work is to endow the well-known drone simulator, AirSim, with a cinematic camera as well as extended its API to control all of its parameters in real time, including various filming lenses and common cinematographic properties. In this paper, we detail the implementation of our AirSim modification, CinemAirSim, present examples that illustrate the potential of the new tool, and highlight the new research opportunities that the use of cinematic cameras can bring to research in robotics and control. https://github.com/ppueyor/CinematicAirSim
BibTeX
@inproceedings{pueyo2020cinemairsim,
  author = {Pablo Pueyo and Eric Cristofalo and Eduardo Montijano and Mac Schwager},
  title = {{CinemAirSim: A Camera-Realistic Robotics Simulator for Cinematographic Purposes}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2020},
  doi = {10.1109/iros45743.2020.9341066},
}
Figure