TFS-NeRF: Template-Free NeRF for Semantic 3D Reconstruction of Dynamic Scene

Sandika Biswas, Qianyi Wu, Biplab Banerjee, Hamid Rezatofighi

TL;DR

TFS-NeRF introduces a template-free, highly efficient 3D semantic NeRF framework designed to reconstruct dynamic scenes featuring complex interactions between two entities from sparse or single-view RGB videos. By replacing computationally heavy iterative root-finding methods with an Invertible Neural Network (INN) for Linear Blend Skinning (LBS) prediction, it achieves accurate, semantically separable 3D geometries with significantly faster training convergence.

What Problem Does This Paper Address?

Reconstructing 3D geometry from dynamic real-world scenes remains a difficult challenge in computer vision. Existing dynamic Neural Radiance Field (NeRF) methods typically fall into two categories, both with major drawbacks:

  1. Entity-specific template models: Methods focusing on humans often rely on parametric priors like SMPL. While effective for humans, they fail to generalize to arbitrary rigid or deformable objects (e.g., animals, boxes, or tools) and struggle to capture fine details like clothing or hair dynamics.
  2. Template-free generic models: These bypass structural priors but usually require extra supervision inputs like depth maps or high-quality optical flow. Advanced template-free methods that utilize forward Linear Blend Skinning (LBS) to map canonical shapes to deformed spaces suffer from massive training bottlenecks. They treat the inverse mapping as a numerical root-finding problem (e.g., using Broyden’s method), requiring expensive matrix inversions at every optimization step.

Furthermore, prior template-free methods focus almost exclusively on single-entity reconstruction, failing to separate distinct interacting elements (like a human handling an object) when severe occlusions occur.

Methods

TFS-NeRF proposes an end-to-end architecture that naturally disentangles the motions of interacting entities without relying on rigid 3D templates.

Overview of the TFS-NeRF system architecture

The pipeline consists of four core stages:

  • Semantic-Aware Ray Sampling: Using 2D semantic masks from the input video, the system shoots distinct sets of rays to sample 3D points for the deformable and non-deformable entities. To handle spatial overlap and occlusions, points are encoded based on their proximity to each entity's 3D skeleton.
  • INN-based View-to-Canonical Warping: Instead of solving the forward LBS equation via slow iterative root-finding, the authors employ an Invertible Neural Network (INN). The INN learns a bijective mapping between the deformed view space and the canonical space. To accelerate convergence, the view space points are pre-initialized using a skeleton-based transformation matrix before passing into the INN.
Figure 3: Overview of the transformation from view space to canonical space.
  • Independent SDF Modules: Once warped into the canonical space, the disentangled points are fed into two independent Signed Distance Field (SDF) networks—one for each entity—to predict clean, semantically isolated geometries.
  • Composite RGB Rendering: A unified RGB prediction network takes the canonical points, features, normals, and posed skeletons from both branches to render the final composite scene, optimized jointly via a combination of reconstruction, cycle-consistency, pose, and skinning weight losses.

Datasets

The authors evaluated their approach across multiple benchmarks representing single-entity movements and multi-entity interactions:

  • BEHAVE: A dataset featuring complex human-object interactions with large motions.
  • HO3D-V3: A benchmark containing hand-object interactions.
  • ZJU-MoCap: Used to benchmark single-human surface reconstruction performance.
  • Synthetic Animal Dataset: A dataset introduced by TAVA to evaluate arbitrary deformable animal reconstructions.

Key Insights and Ideas

  • Bypassing the Root-Finding Bottleneck: The core "aha moment" is leveraging the mathematical properties of INNs to learn the bijective mapping between view and canonical spaces. This avoids the iterative matrix-vector multiplications and inversions typical of Broyden's method, drastically cutting down per-frame computation time.
  • Natural Disentanglement via Dual Heads: Rather than predicting a single density field and attempting to segment it post-hoc with a semantic loss, TFS-NeRF splits the architecture early into per-entity deformation and SDF networks. This structural separation ensures that supervision from an individual entity's observation only updates its respective geometry module, making the framework robust to severe occlusions.

Key Results

TFS-NeRF demonstrates substantial quantitative and qualitative improvements over state-of-the-art generic dynamic NeRFs (such as Tensor4D, NDR, HyperNeRF, and D-NeRF):

  • Reconstruction Accuracy: On the challenging BEHAVE dataset, TFS-NeRF achieved a holistic scene Chamfer Distance of 2.431 cm, significantly outperforming the next-best baseline (NDR + ResField at 3.399 cm).
  • Semantic Separation: It successfully separated human and object geometries during close interactions, yielding a human reconstruction Chamfer Distance of 1.812 cm compared to over 3.8 cm for competing models.
  • Efficiency Gains: The INN formulation reduced per-frame optimization time to 0.3 seconds, a 3× speedup compared to the 0.9 seconds required by the standard Broyden-based LBS formulation.

Limitations and Future Work

While highly effective for dual-entity interactions, the framework currently requires separate networks for each entity. This architecture does not scale efficiently to complex environments containing numerous interacting objects. The authors suggest that integrating an explicit occlusion map could be a promising direction to handle multi-entity scaling. Additionally, the system remains dependent on receiving reasonably accurate 3D skeleton inputs to properly constrain the deformation space; highly inaccurate pose estimations degrade the final reconstruction quality.

Summary updated

Abstract

Despite advancements in Neural Implicit models for 3D surface reconstruction, handling dynamic environments with interactions between arbitrary rigid, non-rigid, or deformable entities remains challenging. The generic reconstruction methods adaptable to such dynamic scenes often require additional inputs like depth or optical flow or rely on pre-trained image features for reasonable outcomes. These methods typically use latent codes to capture frame-by-frame deformations. Another set of dynamic scene reconstruction methods, are entity-specific, mostly focusing on humans, and relies on template models. In contrast, some template-free methods bypass these requirements and adopt traditional LBS (Linear Blend Skinning) weights for a detailed representation of deformable object motions, although they involve complex optimizations leading to lengthy training times. To this end, as a remedy, this paper introduces TFS-NeRF, a template-free 3D semantic NeRF for dynamic scenes captured from sparse or single-view RGB videos, featuring interactions among two entities and more time-efficient than other LBS-based approaches. Our framework uses an Invertible Neural Network (INN) for LBS prediction, simplifying the training process. By disentangling the motions of interacting entities and optimizing per-entity skinning weights, our method efficiently generates accurate, semantically separable geometries. Extensive experiments demonstrate that our approach produces high-quality reconstructions of both deformable and non-deformable objects in complex interactions, with improved training efficiency compared to existing methods.
BibTeX
@inproceedings{biswas2024tfs,
  author = {Sandika Biswas and Qianyi Wu and Biplab Banerjee and Hamid Rezatofighi},
  title = {{TFS-NeRF: Template-Free NeRF for Semantic 3D Reconstruction of Dynamic Scene}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=UPxFYvHsyN},
}
Figure