Introduction
Capturing the perfect tourist photo is a common challenge. Often, the ideal composition is hindered by crowds or suboptimal camera positioning, leading to images where the subject is poorly framed or the landmark is partially obscured. While traditional "Neural Rendering in the Wild" (NRiW) techniques excel at reconstructing static landmarks from vast image collections, they typically treat human subjects as transient noise to be removed. Conversely, Multiplane Imagery (MPI) methods allow for slight perspective shifts but fail to maintain realism during significant camera movements.
This paper introduces a novel framework that bridges this gap, enabling the synthesis of photo-realistic novel views from a single tourist photo. By explicitly modeling both the human subject and the landmark background, the system allows for substantial camera motion while preserving the identity of the person and the structural integrity of the scene.
Methodology
The proposed pipeline treats the human subject and the architectural background as distinct 3D entities, merging them into a unified representation for neural rendering.
- Human Digitization & Localization: The process begins by segmenting the person using Mask R-CNN and DeepLab. A detailed 3D mesh of the individual is then reconstructed from the single 2D image using PIFu (Pixel-aligned Implicit Function). To ensure correct spatial placement, MonoDepth2 provides an initial estimate of the subject's distance from the camera.
- Background Modeling: Unlike the subject, the background is reconstructed using Structure-from-Motion (COLMAP) applied to thousands of publicly available internet photos. This creates a high-fidelity 3D point cloud of the landmark (e.g., the Trevi Fountain).
- Deep Buffer Generation: The human mesh and background point cloud are combined into a "deep buffer." This intermediate representation stores essential data such as depth, color (albedo), and semantic labels.
- Neural Rendering: A deep neural network translates the 3D buffer into a final image. This stage is critical for filling in missing data and smoothing the transition between the mesh and point cloud.
- Lighting Estimation: To achieve seamless integration, the system estimates the lighting on the human subject using Spherical Harmonics, matching it to the environmental lighting of the background.
Key Insights
- Hybrid 3D Representations: The authors utilize a structured mesh for the human subject and an unstructured point cloud for the complex geometry of landmarks. This hybrid approach optimizes the neural renderer's ability to handle different types of visual data.
- Transient Object Management: By explicitly modeling the person rather than filtering them out, the framework can re-project the individual into new perspectives accurately.
- Appearance Transfer: The decoupling of lighting and geometry allows for "teleportation" effects, where a subject can be moved between different landmarks (e.g., from the Trevi Fountain to the Brandenburg Gate) with automated lighting adjustments.
Results and Evaluation
The framework demonstrates significant qualitative success in rendering humans from wide viewing angles that were never captured in the original source image.
Quantitatively, the model achieved a PSNR of 12.38 and a VGG Perceptual Loss of 0.484. While these metrics may appear modest compared to standard image synthesis tasks, the authors emphasize that this is a pioneering effort in a domain where no direct ground truth exists for novel views of specific individuals in these complex environments. The lighting estimator, in particular, showed a high degree of accuracy in capturing directional light patterns, which is vital for preventing the "cardboard cutout" effect.
Limitations and Future Work
Despite its success, the system has several limitations. The current 3D reconstruction method (PIFu) requires a full-body view of the subject, making it less effective for waist-up portraits. Additionally, errors in monocular depth estimation can occasionally lead to scaling issues, where the tourist appears disproportionately large or small relative to the landmark.
Future research aims to address these issues by incorporating human "rigging." This would allow the system to not only change the camera angle but also modify the person's pose, transforming a static "statue" into a dynamic participant in the synthesized scene.
Summary updated