Abstract
Generating cinematic camera tours in 3D environments is a complex task traditionally requiring manual artistry. We present GAIT, a Deep Reinforcement Learning (DRL) framework that automates this process by optimizing for visual aesthetics, temporal smoothness, and spatial diversity. By leveraging a neural aesthetics estimator and a distributed training architecture, GAIT produces professional-quality tours in photorealistic indoor scenes, significantly outperforming traditional optimization baselines in user preference studies.
Introduction
Creating compelling virtual tours for digital twins—such as real estate listings or architectural visualizations—is a labor-intensive process. Traditionally, artists must manually define camera paths and keyframes, a method that is both time-consuming and difficult to scale. Automating this process introduces several technical hurdles: the high-dimensional search space of 5D camera movement (3D position plus yaw and pitch), the subjective nature of "beauty," and the tendency for optimization algorithms to produce jittery paths or get "stuck" at a single optimal viewpoint.
Methodology
GAIT addresses these challenges by framing tour generation as a reinforcement learning problem. The system utilizes state-of-the-art DRL algorithms, specifically DrQ-v2 and CURL, to learn navigation policies directly from visual observations.
System Architecture
To handle the computational demands of rendering high-quality 3D scenes during training, GAIT employs a distributed architecture. This setup separates data collection from policy updates, allowing for high-throughput training.
Multi-Objective Reward Function
The core of GAIT’s success lies in its composite reward function, which balances three competing objectives:
- Aesthetics ($r^A$): A pre-trained neural network scores each frame based on learned aesthetic principles, replacing rigid heuristic rules.
- Temporal Smoothness ($r^S$): To ensure a cinematic feel, the agent is penalized for abrupt changes in velocity. Interestingly, this penalty also prevents the camera from remaining static, as zero-velocity actions are discouraged.
- Diversity Regularization ($r^D$): To ensure the tour explores the entire environment, "exclusion poses" are used to penalize the agent for revisiting previously seen areas, forcing the discovery of new aesthetic viewpoints.
Experimental Results
The researchers evaluated GAIT using the Habitat-Sim framework and the Replica dataset, which contains photorealistic scans of real-world indoor spaces.
Performance and Efficiency
By utilizing an 8-GPU distributed setup, GAIT achieves a throughput of approximately 246 FPS. This efficiency allows the system to converge on a complex, scene-specific navigation policy in roughly 3.5 hours.
Visual Quality and Robustness
User studies indicate a strong preference for GAIT’s output. In comparisons against traditional methods like Evolutionary Strategies (CMA-ES) and Model Predictive Control (MPC), up to 93.3% of participants favored the tours generated by GAIT. Furthermore, the agent demonstrates significant robustness, successfully navigating toward high-aesthetic regions regardless of its initial starting position in the room.
Conclusion and Future Work
GAIT represents a significant advancement in automated cinematography for virtual environments. By combining neural aesthetic scoring with DRL-based navigation, it eliminates the need for manual pathing while maintaining high visual standards. Future research will focus on improving generalization across different scenes without retraining and extending the framework to handle dynamic environments containing moving objects.
Summary updated