Abstract
Dynamic 3D scene reconstruction from multi-view video is a critical challenge for immersive technologies like VR and autonomous driving. While 3D Gaussian Splatting (3DGS) has set new benchmarks for static scenes, its application to dynamic environments often suffers from motion blur due to the limitations of single-network deformation fields. We introduce MAPo (Motion-Aware Partitioning), a strategy that intelligently segments scenes into specialized temporal windows. By identifying high-motion regions and assigning them dedicated deformation networks while freezing static areas, MAPo achieves state-of-the-art rendering quality and efficiency.
Introduction
The reconstruction of dynamic 3D scenes from multi-view video is often considered a "holy grail" for spatial computing. Recent advances in 3D Gaussian Splatting (3DGS) have revolutionized static scene reconstruction due to its high speed and fidelity. However, extending 3DGS to moving scenes typically relies on a "deformation field"—a single neural network that attempts to predict the movement of every point in the scene over time. This "one-size-fits-all" approach frequently leads to two major issues: temporal averaging (motion blur) and redundant computation for static elements like tables or walls.
Methodology
MAPo addresses these challenges through a dynamic score-based partitioning strategy consisting of three main pillars:
Dynamic Scoring and Static Partitioning
During training, the system monitors the historical positions of every 3D Gaussian. It calculates a "dynamic score" based on maximum displacement and position variance. Gaussians with low scores are identified as static and "frozen," bypassing the deformation network entirely. This optimization significantly boosts rendering speed and reduces training overhead.
Recursive Temporal Partitioning
For regions with high dynamic scores, MAPo employs a "divide and conquer" strategy. These Gaussians are recursively split into temporal sub-segments, each assigned a specialized, duplicated deformation network. This allows the model to focus on short bursts of intense, high-frequency motion—such as a hand waving or a spray gun firing—rather than attempting to generalize across the entire video sequence.
Cross-Frame Consistency
To prevent visual artifacts or "seams" at the boundaries of temporal segments, we introduce a Cross-Frame Consistency Loss ($L_{cross}$). This loss function constrains adjacent temporal segments to produce identical, sharp renderings at the moment of transition, ensuring temporal smoothness.
Experimental Results
We evaluated MAPo on the N3DV and Meet Room benchmarks, featuring complex human actions like "flame salmon" and "coffee" captured by multi-camera arrays. MAPo consistently outperformed existing state-of-the-art methods such as 4DGaussians and E-D3DGS.
On the N3DV dataset, MAPo achieved a PSNR of 31.33, a significant improvement over the previous best of ~30.8. Qualitative analysis shows that MAPo successfully recovers fine details—such as the texture of a moving hand—that other methods render as a blurry mess. Despite the use of multiple networks, MAPo maintains real-time performance, rendering at 75-92 FPS on an RTX A6000 with a compact storage footprint of 49-65 MB.
Conclusion
MAPo demonstrates that specialization is superior to generalization in dynamic scene reconstruction. By partitioning the scene based on motion intensity and temporal windows, the model avoids the "averaging trap" inherent in single-network deformation fields. This approach provides a scalable and efficient path toward high-fidelity, real-time dynamic 3D reconstruction.
Summary updated