Introduction
Dynamic scene reconstruction aims to create navigable 4D digital twins from video sequences. Recent advancements in 3D Gaussian Splatting (3DGS) have significantly improved this field. However, existing methods typically require high-frame-rate (dense) video data. When provided with sparse frames—common in low-FPS security footage or stuttering mobile uploads—these models often suffer from "geometric collapse," where reconstructed objects lose structural integrity and detail.
Methodology: Anchoring to Texture
Sparse4DGS addresses the challenge of sparse-frame reconstruction by leveraging high-frequency texture as a geometric guide. The core insight is that texture richness correlates with structural importance, especially when temporal data is limited.
Texture Intensity (TI) Gaussian Field
The framework computes 2D Texture Intensity maps for each input frame using Sobel operators to detect edges and gradients. This TI value is embedded as an attribute within each 3D Gaussian, allowing the model to prioritize high-detail regions during reconstruction.
Texture-Aware Deformation Regularization (TADR)
To maintain structural consistency during motion, Sparse4DGS employs a depth-texture alignment loss. Unlike standard L1 loss, it uses the Pearson Correlation Coefficient (PCC) to align the texture of rendered depth maps with predicted mono-depth maps. This ensures that geometric "edges" in the depth map correspond accurately to visual texture.
Texture-Aware Canonical Optimization (TACO)
During the optimization of the static (canonical) scene, the authors use Stochastic Gradient Langevin Dynamics (SGLD) to inject "smart noise." Gaussians in texture-poor areas receive higher perturbation, encouraging them to converge into texture-rich regions where they contribute more to the scene's structural fidelity.
Experiments and Results
Sparse4DGS was evaluated on benchmarks including NeRF-Synthetic, NeRF-DS, HyperNeRF, and a new iPhone-4D dataset. The method consistently outperforms state-of-the-art techniques like Deformable3DGS and 4DGaussians across various metrics.
On the NeRF-DS dataset with only 20 input frames, Sparse4DGS achieved a PSNR of 22.34, compared to 20.81 for Deformable3DGS. Qualitatively, as shown in the comparison above, Sparse4DGS avoids the blurring and geometric smearing seen in competing methods, maintaining sharp details even at extremely low frame rates. In real-world tests downsampled to 5 FPS, the model maintains high-quality novel view synthesis, demonstrating its readiness for practical applications where high-speed capture is unavailable.
Conclusion
Sparse4DGS represents a significant step forward in making 4D reconstruction robust to sparse data. By treating texture as a primary geometric prior and employing texture-aware regularization and optimization, it enables high-fidelity dynamic scene synthesis from limited temporal information. Future work may involve integrating generative priors to further enhance detail in ultra-sparse settings.
Summary updated