D-TensoRF: Tensorial Radiance Fields for Dynamic Scenes

Hankyu Jang, Daeyoung KimarXiv 2022 · OpenReview

TL;DR

This paper introduces D-TensoRF, a novel method for efficiently reconstructing and rendering dynamic 3D scenes. By extending the Tensorial Radiance Fields (TensoRF) approach to handle time, D-TensoRF achieves state-of-the-art results in novel view synthesis for moving objects with significantly reduced training times and memory footprints compared to existing methods, all without relying on computationally expensive deformation networks.

What Problem Does This Paper Address?

Reconstructing 3D scenes from images has seen remarkable progress with Neural Radiance Fields (NeRF). However, extending NeRF to dynamic scenes – those with moving objects or changing topologies – has been challenging. Existing methods often resort to adding a "deformation network" to map points from a canonical space to their current positions over time. While this allows for dynamic scene modeling, these deformation networks add significant computational overhead, slowing down both training and rendering. The goal here is to model dynamic scenes effectively and efficiently, capturing temporal changes without the performance penalty of deformation networks.

Methods

The core idea is to represent a dynamic scene's radiance field as a 5D tensor, where the dimensions correspond to spatial coordinates (X, Y, Z) and time. This 5D tensor is then decomposed into lower-rank components to achieve a compact representation.

  • Tensorial Representation: The dynamic scene is modeled as a 5D tensor representing a 4D grid (X, Y, Z, time) with multi-channel features.
  • Tensor Decomposition: Similar to the static TensoRF, the paper employs tensor decomposition techniques to factorize this 5D tensor.
  • CP Decomposition: This classic method factorizes the tensor into a sum of rank-one tensor components, essentially outer products of vectors.
  • Matrix-Matrix (MM) Decomposition (Novel): The paper introduces a new MM decomposition. Instead of just vectors, it uses matrix factors that combine two modes (e.g., X and Y, or X and time). This approach is less compact than CP decomposition but significantly reduces computational overhead and training time.
  • Smoothing Regularization: To account for the temporal dependency (how scenes at adjacent times are related), a smoothing regularization term is applied along the time axis of the decomposed factors. This encourages consistency between frames over time.
  • Hybrid Representation: D-TensoRF is a hybrid method, using an explicit 4D grid (the decomposed tensor) and a small neural network (MLP) for view-dependent color prediction. Crucially, it avoids explicit deformation networks.

Datasets

The paper evaluates D-TensoRF on the extended Synthetic NeRF dataset provided by D-NeRF (Pumarola et al., 2020). This dataset contains eight scenes with moving objects, captured at different times.

Key Insights and Ideas

  • Dynamic Scenes as 5D Tensors: The insight that a dynamic scene's radiance can be elegantly represented as a 5D tensor opens the door for efficient tensor decomposition methods.
  • MM Decomposition for Efficiency: The novel Matrix-Matrix decomposition offers a compelling trade-off: slightly increased model size compared to CP decomposition but drastically reduced training time and computational cost. This is a key enabler for practical dynamic scene modeling.
  • Temporal Dependency is Crucial: Simply extending a static model to include time isn't enough. The explicit smoothing regularization is vital for capturing the smooth transitions and relationships between consecutive frames, leading to better temporal consistency.
  • No Deformation Network Needed: The success of D-TensoRF in modeling dynamic scenes without a deformation network is a significant achievement, directly addressing the performance bottleneck of prior work.

Key Results

  • Competitive Rendering Quality: D-TensoRF (both CP and MM variants) achieves quantitatively and qualitatively competitive rendering results compared to state-of-the-art dynamic scene modeling methods.
  • State-of-the-Art LPIPS: The models consistently achieve excellent LPIPS scores, indicating superior perceptual quality, often outperforming other methods even when PSNR is slightly lower.
  • Drastically Reduced Training Time: D-TensoRF-MM achieves impressive 8-minute training times, while D-TensoRF-CP takes around 39-42 minutes for comparable results, both well within an hour. This is a significant improvement over methods taking tens of hours.
  • Compact Memory Footprint: D-TensoRF-CP achieves a remarkably small model size of 1.8MB, significantly smaller than previous dynamic scene methods. D-TensoRF-MM uses 10.8MB, still very competitive.
  • Sharpness: D-TensoRF models tend to produce sharper images compared to many existing methods, especially when motion is smooth.

Limitations and Future Work

  • Large Motions: The paper acknowledges that the Gaussian kernel used for smoothing regularization might not perfectly capture temporal dependencies when there are very large motions between adjacent frames. This can lead to smudged results in such cases.
  • Bounded Scenes: Similar to TensoRF, D-TensoRF is currently limited to modeling bounded scenes.
  • Future Directions:
    • Developing more sophisticated weight functions for regularization that can better handle large temporal movements.
  • Extending the method to unbounded scenes, potentially by modeling background and foreground objects separately or exploring other approaches.

Summary updated

Abstract

Neural radiance field (NeRF) attracts attention as a promising approach to reconstructing the 3D scene. As NeRF emerges, subsequent studies have been conducted to model dynamic scenes, which include motions or topological changes. However, most of them use an additional deformation network, slowing down the training and rendering speed. Tensorial radiance field (TensoRF) recently shows its potential for fast, high-quality reconstruction of static scenes with compact model size. In this paper, we present D-TensoRF, a tensorial radiance field for dynamic scenes, enabling novel view synthesis at a specific time. We consider the radiance field of a dynamic scene as a 5D tensor. The 5D tensor represents a 4D grid in which each axis corresponds to X, Y, Z, and time and has 1D multi-channel features per element. Similar to TensoRF, we decompose the grid either into rank-one vector components (CP decomposition) or low-rank matrix components (newly proposed MM decomposition). We also use smoothing regularization to reflect the relationship between features at different times (temporal dependency). We conduct extensive evaluations to analyze our models. We show that D-TensoRF with CP decomposition and MM decomposition both have short training times and significantly low memory footprints with quantitatively and qualitatively competitive rendering results in comparison to the state-of-the-art methods in 3D dynamic scene modeling.
BibTeX
@misc{jang2022d,
  author = {Hankyu Jang and Daeyoung Kim},
  title = {{D-TensoRF: Tensorial Radiance Fields for Dynamic Scenes}},
  year = {2022},
  eprint = {2212.02375},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2212.02375},
}
Figure