Learning Dynamic View Synthesis With Few RGBD Cameras

Henry FuchsarXiv 2022 · DOI · arXiv

TL;DR

This paper introduces a novel method for synthesizing realistic free-viewpoint videos of dynamic indoor scenes using only a few commodity RGBD cameras. By addressing the challenges of noisy depth data and enforcing spatial-temporal consistency, their approach achieves significant improvements in visual quality and stability compared to existing methods, without requiring per-scene optimization or prior human models.

What Problem Does This Paper Address?

Creating realistic free-viewpoint videos of dynamic scenes, where objects and people are in motion, is a challenging task. Current deep learning methods often struggle with this due to several limitations:

  • Reliance on prior models: Many methods require pre-existing models like SMPL for human bodies, which limits their generalizability to new subjects or objects.
  • Heavy preprocessing: Some approaches demand extensive pre-processing of input data, making them impractical for real-time applications.
  • Per-scene optimization: Many state-of-the-art methods require optimizing the model for each new scene or video, which is computationally expensive and time-consuming.
  • Inaccurate depth data: Commodity RGBD cameras, while convenient, provide depth measurements that are often noisy, unstable, and incomplete. This leads to artifacts like distortions, flickering, and ghosting in the synthesized views.

This paper aims to overcome these limitations by developing a system that can synthesize dynamic novel views using a sparse set of readily available RGBD cameras, without per-scene optimization or heavy preprocessing, and by explicitly handling the inaccuracies of depth data.

Methods

The proposed method leverages a sparse set of synchronized and calibrated RGBD cameras to synthesize dynamic novel views. The core components are:

  • Feature Point Cloud Generation: RGB images are encoded into feature maps, which are then "lifted" into 3D feature point clouds using the depth information from the RGBD sensors.
  • Regional Depth-Inpainting: To address the incomplete and noisy depth maps from RGBD cameras, a novel Regional Depth-Inpainting module is introduced. Instead of a naive global alignment, it identifies regions with missing depth and performs local alignment using estimated depth maps, significantly reducing artifacts.
  • Cycle Reconstruction Consistency (CRC): This module enforces spatial-temporal consistency by requiring the model to reconstruct the original input source images from the synthesized novel views. This helps to reduce artifacts that are inconsistent with the input.
  • Temporal Stabilization (TS): To further improve temporal consistency and reduce flickering, a temporal-aware neural renderer is used. It implicitly aggregates information from the current and previous frames, stabilizing the synthesis process.

These components are integrated into a neural renderer that generates the final novel view images.

Datasets

  • Human-Things Interaction (HTI) Dataset: The authors introduce a new dataset specifically for dynamic novel view synthesis. It comprises 43 multi-view RGBD video sequences capturing complex interactions between human subjects and objects in 7 different indoor environments. This dataset is designed to facilitate the training and evaluation of generalizable dynamic view synthesis models.

Key Insights and Ideas

  • RGBD cameras are sufficient: The paper demonstrates that a sparse setup of commodity RGBD cameras can be effective for dynamic novel view synthesis, removing the need for more complex or expensive capture systems.
  • Depth noise is a critical challenge: The authors highlight that the inaccuracies in depth measurements from RGBD cameras are a primary source of artifacts and that explicitly addressing this is crucial for high-quality synthesis.
  • Spatial-temporal consistency is key: Enforcing consistency across both space (different views) and time (consecutive frames) is essential for producing stable and artifact-free videos.
  • Regional depth inpainting is more effective: A localized approach to depth completion, rather than a global one, significantly improves the handling of noisy and incomplete depth data.
  • Reconstruction loss aids artifact reduction: The Cycle Reconstruction Consistency loss proves effective in reducing artifacts by ensuring the synthesized views are consistent with the original input.

Key Results

  • Quantitative Performance: The proposed method outperforms baseline methods like FVS, NSFF, and DynamicNeRF across various metrics (PSNR, SSIM, LPIPS). For instance, in action generalization, it achieves +0.564 dB PSNR, +1.900% SSIM, and -11.842% LPIPS loss compared to FVS. In complete generalization, it shows even larger margins, with +1.211 dB PSNR, +4.589% SSIM, and -19.813% LPIPS loss over NSFF and DynamicNeRF.
  • Qualitative Improvements: The model significantly reduces distortions, flickering, and ghosting artifacts. It produces sharper images, more accurate facial details, and better reconstruction of fine structures (e.g., clothing folds, text on objects). The Regional Depth-Inpainting module successfully recovers details like a plastic bag that are missed by baselines due to missing depth.
  • Temporal Stability: The Temporal Stabilization module notably improves temporal stability, reducing flickering and producing smoother video sequences, even with unstable depth inputs.
  • Generalization: The approach demonstrates good generalization to new actions, people, and environments without per-scene optimization.

Limitations and Future Work

  • Reliance on accurate depth: The paper acknowledges that the performance is still limited by the inherent quality of the RGBD camera depth measurements.
  • Flickering with low-fidelity cameras: Despite improvements, some flickering artifacts persist, especially with lower-fidelity RGBD cameras like the Kinect V2.
  • Future Work:
    • Utilizing better RGBD cameras (e.g., Azure Kinect) for improved depth quality.
  • Exploring simple background pre-scanning techniques.
  • Investigating depth refinement using visual observations.
  • Improving the speed of the system to enable real-time novel view synthesis for telepresence applications.

Summary updated

Abstract

There have been significant advancements in dynamic novel view synthesis in recent years. However, current deep learning models often require (1) prior models (e.g., SMPL human models), (2) heavy pre-processing, or (3) per-scene optimization. We propose to utilize RGBD cameras to remove these limitations and synthesize free-viewpoint videos of dynamic indoor scenes. We generate feature point clouds from RGBD frames and then render them into free-viewpoint videos via a neural renderer. However, the inaccurate, unstable, and incomplete depth measurements induce severe distortions, flickering, and ghosting artifacts. We enforce spatial-temporal consistency via the proposed Cycle Reconstruction Consistency and Temporal Stabilization module to reduce these artifacts. We introduce a simple Regional Depth-Inpainting module that adaptively inpaints missing depth values to render complete novel views. Additionally, we present a Human-Things Interactions dataset to validate our approach and facilitate future research. The dataset consists of 43 multi-view RGBD video sequences of everyday activities, capturing complex interactions between human subjects and their surroundings. Experiments on the HTI dataset show that our method outperforms the baseline per-frame image fidelity and spatial-temporal consistency. We will release our code, and the dataset on the website soon.
BibTeX
@misc{fuchs2022learning,
  author = {Henry Fuchs},
  title = {{Learning Dynamic View Synthesis With Few RGBD Cameras}},
  year = {2022},
  eprint = {2204.10477},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2204.10477},
}
Figure