TFS-NeRF: Template-Free NeRF for Semantic 3D Reconstruction of Dynamic Scene

Sandika Biswas, Qianyi Wu, Biplab Banerjee, Hamid Rezatofighi

TFS-NeRF is a novel framework designed to reconstruct high-quality 3D geometry of multiple interacting entities—such as a human playing with a ball—from sparse or single-view RGB videos. By replacing slow, iterative optimization with an Invertible Neural Network (INN) and a template-free approach, it achieves semantically separable 3D reconstructions with significantly better training efficiency than previous state-of-the-art methods.

The Challenge of Dynamic Scene Reconstruction

Reconstructing dynamic 3D scenes from standard video is a long-standing goal for AR/VR and robotics. However, current Neural Radiance Field (NeRF) methods face three major hurdles:

  1. Template Dependency: Many models rely on pre-defined templates (like SMPL for humans), limiting their ability to handle arbitrary objects or animals.
  2. Interaction Complexity: When objects interact (e.g., a hand grabbing a bottle), occlusions and complex deformations make it difficult to assign 3D points to specific entities.
  3. Computational Bottlenecks: Existing template-free methods often use Linear Blend Skinning (LBS) to map moving points back to a static "canonical" space. This typically requires slow, iterative mathematical processes (like Broyden’s method) involving expensive matrix inversions, leading to grueling training times.

Methodology: A Template-Free, Semantic Approach

TFS-NeRF introduces several architectural shifts to streamline the reconstruction process and ensure semantic awareness.

The TFS-NeRF pipeline: (A) Semantic-aware ray sampling, (B) Inverse warping via INN, (C) Disentangled SDF modules for different entities, and (D) Volume rendering.

Key Architectural Components

  • Semantic-Aware Ray Sampling: Instead of treating the scene as a single volume, the model uses 2D segmentation masks to sample rays for each entity. It employs "semantic encoding" based on the distance to 3D skeletons to ensure points are correctly assigned even during heavy occlusion.
  • Invertible Neural Networks (INN): This is the core efficiency driver. Rather than using slow iterative solvers to find correspondences between "deformed" (moving) and "canonical" (static) space, TFS-NeRF uses an INN. Because INNs are mathematically bijective, the model learns forward and backward mappings simultaneously without iterative root-finding.
  • Disentangled SDF Modules: The framework uses independent Signed Distance Field (SDF) networks for different entities. This allows the model to output semantically separable geometries—enabling users to "unplug" a reconstructed human from an object.
  • Skeleton-Based Initialization: To assist INN convergence, the model uses a rough 3D skeleton estimated from the video as a starting point for coordinate transformations.

Experimental Results

The model was evaluated on diverse datasets, including BEHAVE (human-object interaction), HO3D-V3 (hand-object interaction), ZJU-MoCap, and a Synthetic Animal Dataset to demonstrate its template-free generalizability.

Qualitative results across different datasets showing (A) input, (B) normal maps, (C) reconstructed geometry, and (D) semantic segmentation.

Performance Highlights

  • Accuracy: On the BEHAVE dataset, TFS-NeRF achieved a Chamfer Distance of 1.812 cm for human reconstruction, nearly doubling the accuracy of the next best baseline (which averaged ~3.9 cm).
  • Semantic Quality: It successfully reconstructed distinct meshes for interacting objects where baseline models often "fused" entities together or produced jagged surfaces.
  • Speed: The INN-based approach provided a 3x speedup in the coordinate transformation step, reducing per-frame processing time from 0.9 seconds to 0.3 seconds.

Discussion and Future Work

By focusing on learning structural relationships (skinning weights) rather than just latent codes, TFS-NeRF produces smoother, more physically plausible surfaces. The natural disentanglement provided by separate SDF heads avoids the "ghosting" artifacts common in neural volumes when objects pass close to one another.

While highly effective, the current architecture requires separate networks for each entity, which may limit scalability in crowded scenes. Additionally, the model remains dependent on reasonably accurate initial 3D skeleton estimations. Future research may explore the integration of occlusion maps to further enhance performance in complex, multi-object environments.

Summary updated

Abstract

Despite advancements in Neural Implicit models for 3D surface reconstruction, handling dynamic environments with interactions between arbitrary rigid, non-rigid, or deformable entities remains challenging. The generic reconstruction methods adaptable to such dynamic scenes often require additional inputs like depth or optical flow or rely on pre-trained image features for reasonable outcomes. These methods typically use latent codes to capture frame-by-frame deformations. Another set of dynamic scene reconstruction methods, are entity-specific, mostly focusing on humans, and relies on template models. In contrast, some template-free methods bypass these requirements and adopt traditional LBS (Linear Blend Skinning) weights for a detailed representation of deformable object motions, although they involve complex optimizations leading to lengthy training times. To this end, as a remedy, this paper introduces TFS-NeRF, a template-free 3D semantic NeRF for dynamic scenes captured from sparse or single-view RGB videos, featuring interactions among two entities and more time-efficient than other LBS-based approaches. Our framework uses an Invertible Neural Network (INN) for LBS prediction, simplifying the training process. By disentangling the motions of interacting entities and optimizing per-entity skinning weights, our method efficiently generates accurate, semantically separable geometries. Extensive experiments demonstrate that our approach produces high-quality reconstructions of both deformable and non-deformable objects in complex interactions, with improved training efficiency compared to existing methods. The code and models will be available on our github page.
BibTeX
@inproceedings{biswas2024tfs,
  author = {Sandika Biswas and Qianyi Wu and Biplab Banerjee and Hamid Rezatofighi},
  title = {{TFS-NeRF: Template-Free NeRF for Semantic 3D Reconstruction of Dynamic Scene}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=UPxFYvHsyN},
}
Figure