TFS-NeRF is a novel framework designed to reconstruct high-quality 3D geometry of multiple interacting entities—such as a human playing with a ball—from sparse or single-view RGB videos. By replacing slow, iterative optimization with an Invertible Neural Network (INN) and a template-free approach, it achieves semantically separable 3D reconstructions with significantly better training efficiency than previous state-of-the-art methods.
The Challenge of Dynamic Scene Reconstruction
Reconstructing dynamic 3D scenes from standard video is a long-standing goal for AR/VR and robotics. However, current Neural Radiance Field (NeRF) methods face three major hurdles:
- Template Dependency: Many models rely on pre-defined templates (like SMPL for humans), limiting their ability to handle arbitrary objects or animals.
- Interaction Complexity: When objects interact (e.g., a hand grabbing a bottle), occlusions and complex deformations make it difficult to assign 3D points to specific entities.
- Computational Bottlenecks: Existing template-free methods often use Linear Blend Skinning (LBS) to map moving points back to a static "canonical" space. This typically requires slow, iterative mathematical processes (like Broyden’s method) involving expensive matrix inversions, leading to grueling training times.
Methodology: A Template-Free, Semantic Approach
TFS-NeRF introduces several architectural shifts to streamline the reconstruction process and ensure semantic awareness.
Key Architectural Components
- Semantic-Aware Ray Sampling: Instead of treating the scene as a single volume, the model uses 2D segmentation masks to sample rays for each entity. It employs "semantic encoding" based on the distance to 3D skeletons to ensure points are correctly assigned even during heavy occlusion.
- Invertible Neural Networks (INN): This is the core efficiency driver. Rather than using slow iterative solvers to find correspondences between "deformed" (moving) and "canonical" (static) space, TFS-NeRF uses an INN. Because INNs are mathematically bijective, the model learns forward and backward mappings simultaneously without iterative root-finding.
- Disentangled SDF Modules: The framework uses independent Signed Distance Field (SDF) networks for different entities. This allows the model to output semantically separable geometries—enabling users to "unplug" a reconstructed human from an object.
- Skeleton-Based Initialization: To assist INN convergence, the model uses a rough 3D skeleton estimated from the video as a starting point for coordinate transformations.
Experimental Results
The model was evaluated on diverse datasets, including BEHAVE (human-object interaction), HO3D-V3 (hand-object interaction), ZJU-MoCap, and a Synthetic Animal Dataset to demonstrate its template-free generalizability.
Performance Highlights
- Accuracy: On the BEHAVE dataset, TFS-NeRF achieved a Chamfer Distance of 1.812 cm for human reconstruction, nearly doubling the accuracy of the next best baseline (which averaged ~3.9 cm).
- Semantic Quality: It successfully reconstructed distinct meshes for interacting objects where baseline models often "fused" entities together or produced jagged surfaces.
- Speed: The INN-based approach provided a 3x speedup in the coordinate transformation step, reducing per-frame processing time from 0.9 seconds to 0.3 seconds.
Discussion and Future Work
By focusing on learning structural relationships (skinning weights) rather than just latent codes, TFS-NeRF produces smoother, more physically plausible surfaces. The natural disentanglement provided by separate SDF heads avoids the "ghosting" artifacts common in neural volumes when objects pass close to one another.
While highly effective, the current architecture requires separate networks for each entity, which may limit scalability in crowded scenes. Additionally, the model remains dependent on reasonably accurate initial 3D skeleton estimations. Future research may explore the integration of occlusion maps to further enhance performance in complex, multi-object environments.
Summary updated