Aesthetics Driven Autonomous Time-Lapse Photography Generation by Virtual and Real Robots

Xiaobo Gao, Qi Kuang, Xin Jin, Bin Zhou, Boyan Dong, Xunyu WangarXiv 2022 · DOI · arXiv

Abstract

Time-lapse photography traditionally requires significant patience and artistic expertise. This paper presents a framework that automates the entire process by integrating 3D scene reconstruction with computational aesthetics. By parameterizing artistic choices—viewfinder, camera movement, and time-warp—and optimizing them via deep-learning-based aesthetic assessment models, the system enables both virtual and real-world robots to capture professional-grade time-lapse videos autonomously.

Introduction

Time-lapse photography is a compelling storytelling medium, yet it presents a steep barrier to entry. It demands extreme physical patience and a refined artistic intuition to predict how light, shadows, and movement will evolve over several hours. While robotic platforms offer the precision and stability required for the "patience" aspect, they typically lack the "eye" for aesthetics necessary to select optimal viewpoints or camera paths. This research bridges the gap by developing a system that "thinks" like a photographer to guide robotic hardware through a multi-stage optimization pipeline.

The Proposed Framework

The proposed system utilizes a two-part pipeline: parameterization and robotic execution.

Extracted figure from page 3 of Aesthetics Driven Autonomous Time-Lapse Photography Generation by Virtual and Real Robots

1. 3D Scene Reconstruction and Simulation

The process begins with Structure-from-Motion (SfM) to create a textured 3D "digital twin" of the target environment from standard images. To ensure the simulation is representative of the real world, dynamic elements such as simulated traffic, pedestrians, and varying lighting conditions are integrated into the model.

2. Parameterization of Aesthetics

The framework decomposes the artistic task into three distinct mathematical categories:

  • Viewfinder ($P_I$): Determines the optimal camera location and orientation based on image aesthetic assessment.
  • Camera Movement ($P_V$): Defines the trajectory and speed of the camera, optimized for video aesthetics.
  • Time-Warp ($P_T$): Selects the specific time range and frame intervals to capture time-lapse-specific aesthetic features.

3. Optimization and Virtual Shooting

Deep-learning-based aesthetic quality assessment (AQA) models are employed to score various parameter combinations. The system iteratively searches for the configuration that maximizes the predicted aesthetic score. A "Virtual Shooting" interface allows users to preview the results and manually adjust parameters if desired, maintaining a human-in-the-loop approach.

4. Robotic Execution

Once optimized, the parameters—including GPS coordinates, pan-tilt angles, and timing—are exported to a physical robotic platform. The robot then executes the planned path in the real world to capture the final image sequences.

Key Insights

  • Digital Twin Rehearsal: Using a 3D model as a rehearsal space allows the system to simulate hours of environmental changes in seconds. This enables the discovery of "golden hour" lighting without requiring the robot to remain on-site for extended periods.
  • Decomposed Optimization: By separating image, video, and time-lapse aesthetics, the framework can apply specialized models to specific sub-problems, balancing framing quality with movement smoothness.
  • Human-AI Collaboration: The inclusion of a virtual shooting interface ensures the system acts as a collaborative tool rather than a "black box," allowing creators to refine AI-generated suggestions.

Experimental Results

The system was evaluated through ablation studies and comparisons with human photographers across 20 distinct test scenarios in simulated environments.

Ablation Study

The ablation study confirmed that the integration of all three parameter sets (Image, Video, and Time) is crucial for high-quality results.

Ablation study results showing the incremental improvement in aesthetic quality scores as more optimization layers are added.

As shown in the table, optimizing for all three factors yielded a quality score of 0.635, a significant improvement over the baseline of 0.508 for random selection.

Human Comparison

In tests against human subjects, the system consistently outperformed amateur photographers (average score 0.555 vs. 0.468). Most notably, the system's performance was found to be comparable to that of professional photographers (who averaged 0.568), indicating that the framework successfully captures expert-level heuristics.

Conclusion and Future Work

While the framework demonstrates significant potential, certain limitations remain. The current simulation of dynamic elements (like traffic) does not fully account for real-world unpredictability, such as sudden weather changes. Additionally, while histogram equalization helps mitigate exposure flicker, lighting transitions remain a complex challenge. Future work will focus on improving real-time adaptation to environmental shifts and addressing the physical constraints of robotic hardware, such as battery life and complex terrain navigation.

Summary updated

Abstract

Time-lapse photography is employed in movies and promotional films because it can reflect the passage of time in a short time and strengthen the visual attraction. However, since it takes a long time and requires the stable shooting, it is a great challenge for the photographer. In this article, we propose a time-lapse photography system with virtual and real robots. To help users shoot time-lapse videos efficiently, we first parameterize the time-lapse photography and propose a parameter optimization method. For different parameters, different aesthetic models, including image and video aesthetic quality assessment networks, are used to generate optimal parameters. Then we propose a time-lapse photography interface to facilitate users to view and adjust parameters and use virtual robots to conduct virtual photography in a three-dimensional scene. The system can also export the parameters and provide them to real robots so that the time-lapse videos can be filmed in the real world. In addition, we propose a time-lapse photography aesthetic assessment method that can automatically evaluate the aesthetic quality of time-lapse video. The experimental results show that our method can efficiently obtain the time-lapse videos. We also conduct a user study. The results show that our system has the similar effect as professional photographers and is more efficient.
BibTeX
@misc{gao2022aesthetics,
  author = {Xiaobo Gao and Qi Kuang and Xin Jin and Bin Zhou and Boyan Dong and Xunyu Wang},
  title = {{Aesthetics Driven Autonomous Time-Lapse Photography Generation by Virtual and Real Robots}},
  year = {2022},
  eprint = {2208.10181},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2208.10181},
}
Figure