GAIT: Generating Aesthetic Indoor Tours with Deep Reinforcement Learning

Desai Xie, Ping Hu, Xin Sun, Sören Pirk, Jianming Zhang, Radomír Mech, Arie E. Kaufman

Abstract

Generating cinematic camera tours in 3D environments is a complex task traditionally requiring manual artistry. We present GAIT, a Deep Reinforcement Learning (DRL) framework that automates this process by optimizing for visual aesthetics, temporal smoothness, and spatial diversity. By leveraging a neural aesthetics estimator and a distributed training architecture, GAIT produces professional-quality tours in photorealistic indoor scenes, significantly outperforming traditional optimization baselines in user preference studies.

Introduction

Creating compelling virtual tours for digital twins—such as real estate listings or architectural visualizations—is a labor-intensive process. Traditionally, artists must manually define camera paths and keyframes, a method that is both time-consuming and difficult to scale. Automating this process introduces several technical hurdles: the high-dimensional search space of 5D camera movement (3D position plus yaw and pitch), the subjective nature of "beauty," and the tendency for optimization algorithms to produce jittery paths or get "stuck" at a single optimal viewpoint.

Methodology

GAIT addresses these challenges by framing tour generation as a reinforcement learning problem. The system utilizes state-of-the-art DRL algorithms, specifically DrQ-v2 and CURL, to learn navigation policies directly from visual observations.

System Architecture

To handle the computational demands of rendering high-quality 3D scenes during training, GAIT employs a distributed architecture. This setup separates data collection from policy updates, allowing for high-throughput training.

The distributed training architecture of GAIT, showing the interaction between multiple Data Workers, a centralized Replay Buffer, and the Update Worker.

Multi-Objective Reward Function

The core of GAIT’s success lies in its composite reward function, which balances three competing objectives:

  1. Aesthetics ($r^A$): A pre-trained neural network scores each frame based on learned aesthetic principles, replacing rigid heuristic rules.
  2. Temporal Smoothness ($r^S$): To ensure a cinematic feel, the agent is penalized for abrupt changes in velocity. Interestingly, this penalty also prevents the camera from remaining static, as zero-velocity actions are discouraged.
  3. Diversity Regularization ($r^D$): To ensure the tour explores the entire environment, "exclusion poses" are used to penalize the agent for revisiting previously seen areas, forcing the discovery of new aesthetic viewpoints.

Experimental Results

The researchers evaluated GAIT using the Habitat-Sim framework and the Replica dataset, which contains photorealistic scans of real-world indoor spaces.

Performance and Efficiency

By utilizing an 8-GPU distributed setup, GAIT achieves a throughput of approximately 246 FPS. This efficiency allows the system to converge on a complex, scene-specific navigation policy in roughly 3.5 hours.

Visual Quality and Robustness

User studies indicate a strong preference for GAIT’s output. In comparisons against traditional methods like Evolutionary Strategies (CMA-ES) and Model Predictive Control (MPC), up to 93.3% of participants favored the tours generated by GAIT. Furthermore, the agent demonstrates significant robustness, successfully navigating toward high-aesthetic regions regardless of its initial starting position in the room.

Generated image sequences and 3D trajectories for three different starting poses, illustrating the smoothness and diversity of the resulting tours.

Conclusion and Future Work

GAIT represents a significant advancement in automated cinematography for virtual environments. By combining neural aesthetic scoring with DRL-based navigation, it eliminates the need for manual pathing while maintaining high visual standards. Future research will focus on improving generalization across different scenes without retraining and extending the framework to handle dynamic environments containing moving objects.

Summary updated

Abstract

Placing and orienting a camera to compose aesthetically meaningful shots of a scene is not only a key objective in real-world photography and cinematography but also for virtual content creation. The framing of a camera often significantly contributes to the story telling in movies, games, and mixed reality applications. Generating single camera poses or even contiguous trajectories either requires a significant amount of manual labor or requires solving high-dimensional optimization problems, which can be computationally demanding and error-prone. In this paper, we introduce GAIT, a framework for training a Deep Reinforcement Learning (DRL) agent, that learns to automatically control a camera to generate a sequence of aesthetically meaningful views for synthetic 3D indoor scenes. To generate sequences of frames with high aesthetic value, GAIT relies on a neural aesthetics estimator, which is trained on a crowed-sourced dataset. Additionally, we introduce regularization techniques for diversity and smoothness to generate visually interesting trajectories for a 3D environment, and to constrain agent acceleration in the reward function to generate a smooth sequence of camera frames. We validated our method by comparing it to baseline algorithms, based on a perceptual user study, and through ablation studies. Code and visual results are available on the project website: https://desaixie.github.io/gait-rl
BibTeX
@inproceedings{xie2023gait,
  author = {Desai Xie and Ping Hu and Xin Sun and Sören Pirk and Jianming Zhang and Radomír Mech and Arie E. Kaufman},
  title = {{GAIT: Generating Aesthetic Indoor Tours with Deep Reinforcement Learning}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  doi = {10.1109/iccv51070.2023.00681},
}
Figure