Novel-View Synthesis of Human Tourist Photos

Jonathan Freer, Kwang Moo Yi, Wei Jiang, Jongwon Choi, Hyung Jin Chang

Introduction

Capturing the perfect tourist photo is a common challenge. Often, the ideal composition is hindered by crowds or suboptimal camera positioning, leading to images where the subject is poorly framed or the landmark is partially obscured. While traditional "Neural Rendering in the Wild" (NRiW) techniques excel at reconstructing static landmarks from vast image collections, they typically treat human subjects as transient noise to be removed. Conversely, Multiplane Imagery (MPI) methods allow for slight perspective shifts but fail to maintain realism during significant camera movements.

This paper introduces a novel framework that bridges this gap, enabling the synthesis of photo-realistic novel views from a single tourist photo. By explicitly modeling both the human subject and the landmark background, the system allows for substantial camera motion while preserving the identity of the person and the structural integrity of the scene.

Methodology

The proposed pipeline treats the human subject and the architectural background as distinct 3D entities, merging them into a unified representation for neural rendering.

Overview of the pipeline: from a single input photo to 3D scene reconstruction and finally neurally rendered novel views.
  1. Human Digitization & Localization: The process begins by segmenting the person using Mask R-CNN and DeepLab. A detailed 3D mesh of the individual is then reconstructed from the single 2D image using PIFu (Pixel-aligned Implicit Function). To ensure correct spatial placement, MonoDepth2 provides an initial estimate of the subject's distance from the camera.
  2. Background Modeling: Unlike the subject, the background is reconstructed using Structure-from-Motion (COLMAP) applied to thousands of publicly available internet photos. This creates a high-fidelity 3D point cloud of the landmark (e.g., the Trevi Fountain).
  3. Deep Buffer Generation: The human mesh and background point cloud are combined into a "deep buffer." This intermediate representation stores essential data such as depth, color (albedo), and semantic labels.
  4. Neural Rendering: A deep neural network translates the 3D buffer into a final image. This stage is critical for filling in missing data and smoothing the transition between the mesh and point cloud.
  5. Lighting Estimation: To achieve seamless integration, the system estimates the lighting on the human subject using Spherical Harmonics, matching it to the environmental lighting of the background.

Key Insights

  • Hybrid 3D Representations: The authors utilize a structured mesh for the human subject and an unstructured point cloud for the complex geometry of landmarks. This hybrid approach optimizes the neural renderer's ability to handle different types of visual data.
  • Transient Object Management: By explicitly modeling the person rather than filtering them out, the framework can re-project the individual into new perspectives accurately.
  • Appearance Transfer: The decoupling of lighting and geometry allows for "teleportation" effects, where a subject can be moved between different landmarks (e.g., from the Trevi Fountain to the Brandenburg Gate) with automated lighting adjustments.

Results and Evaluation

The framework demonstrates significant qualitative success in rendering humans from wide viewing angles that were never captured in the original source image.

A neurally rendered novel view showing the synthesized background and the re-projected human subject with consistent lighting.

Quantitatively, the model achieved a PSNR of 12.38 and a VGG Perceptual Loss of 0.484. While these metrics may appear modest compared to standard image synthesis tasks, the authors emphasize that this is a pioneering effort in a domain where no direct ground truth exists for novel views of specific individuals in these complex environments. The lighting estimator, in particular, showed a high degree of accuracy in capturing directional light patterns, which is vital for preventing the "cardboard cutout" effect.

Limitations and Future Work

Despite its success, the system has several limitations. The current 3D reconstruction method (PIFu) requires a full-body view of the subject, making it less effective for waist-up portraits. Additionally, errors in monocular depth estimation can occasionally lead to scaling issues, where the tourist appears disproportionately large or small relative to the landmark.

Future research aims to address these issues by incorporating human "rigging." This would allow the system to not only change the camera angle but also modify the person's pose, transforming a static "statue" into a dynamic participant in the synthesized scene.

Summary updated

Abstract

We present a novel framework for performing novel-view synthesis on human tourist photos. Given a tourist photo from a known scene, we reconstruct the photo in 3D space through modeling the human and the background independently. We generate a deep buffer from a novel viewpoint of the reconstruction and utilize a deep network to translate the buffer into a photo-realistic rendering of the novel view. We additionally present a method to relight the renderings, allowing for relighting of both human and background to match either the provided input image or any other. The key contributions of our paper are: 1) a framework for performing novel view synthesis on human tourist photos, 2) an appearance transfer method for relighting of humans to match synthesized backgrounds, and 3) a method for estimating lighting properties from a single human photo. We demonstrate the proposed framework on photos from two different scenes of various tourists.
BibTeX
@inproceedings{freer2022novel,
  author = {Jonathan Freer and Kwang Moo Yi and Wei Jiang and Jongwon Choi and Hyung Jin Chang},
  title = {{Novel-View Synthesis of Human Tourist Photos}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2022},
  doi = {10.1109/wacv51458.2022.00093},
}
Figure