Abstract
3D Gaussian Splatting (3DGS) has made significant strides in scene representation and neural rendering, with intense efforts focused on adapting it for dynamic scenes. Despite delivering remarkable rendering quality and speed, existing methods struggle with storage demands and representing complex real-world motions. To tackle these issues, we propose MoDec-GS, a memory-efficient Gaussian splatting framework designed for reconstructing novel views in challenging scenarios with complex motions. We introduce Global-to-Local Motion Decomposition (GLMD) to effectively capture dynamic motions in a coarse-to-fine manner. This approach leverages Global Canonical Scaffolds (Global CS) and Local Canonical Scaffolds (Local CS), extending static Scaffold representation to dynamic video reconstruction. For Global CS, we propose Global Anchor Deformation (GAD) to efficiently represent global dynamics along complex motions, by directly deforming the implicit Scaffold attributes which are anchor position, offset, and local context features. Next, we finely adjust local motions via the Local Gaussian Deformation (LGD) of Local CS explicitly. Additionally, we introduce Temporal Interval Adjustment (TIA) to automatically control the temporal coverage of each Local CS during training, allowing MoDec-GS to find optimal interval assignments based on the specified number of temporal segments. Extensive evaluations demonstrate that MoDec-GS achieves an average 70% reduction in model size over state-of-the-art methods for dynamic 3D Gaussians from real-world dynamic videos while maintaining or even improving rendering quality.
Introduction
3D Gaussian Splatting (3DGS) has revolutionized real-time 3D reconstruction, but extending it to dynamic scenes remains a significant challenge. Current dynamic 3DGS methods often face a trade-off: they either require massive storage to track Gaussian trajectories over time or struggle to represent complex, long-duration motions, leading to blurry reconstructions. Furthermore, many existing models rely on fixed time intervals or pre-computed optical flow, which can be computationally expensive and inaccurate for complex real-world scenarios. MoDec-GS addresses these limitations by creating a framework that is both compact and expressive without requiring external motion data.
Methodology
The core of MoDec-GS is a coarse-to-fine motion modeling strategy built upon an anchor-based representation (Scaffold-GS). The framework introduces three primary innovations to handle complex dynamics efficiently.
Global-to-Local Motion Decomposition (GLMD)
Instead of modeling all motion simultaneously, MoDec-GS decomposes it into two distinct stages:
- Global Anchor Deformation (GAD): A compact "global hexplane" deforms sparse anchor points to capture large-scale movements (e.g., a hand moving across a frame). By deforming the implicit Scaffold attributes—anchor position, offset, and local context features—the model efficiently represents global dynamics.
- Local Gaussian Deformation (LGD): Once anchors are positioned, a shared "local hexplane" handles fine-grained, subtle movements of individual Gaussians relative to those anchors.
Temporal Interval Adjustment (TIA)
To optimize model capacity, TIA provides an automated "re-balancing" mechanism. During training, the model monitors positional gradients to identify time segments where motion is difficult to represent. It adaptively shrinks time intervals for high-motion segments and expands them for static ones, ensuring that the model's parameters are allocated where they are most needed.
Results and Discussion
MoDec-GS was evaluated on several challenging monocular video datasets, including Dycheck-iPhone, HyperNeRF, and the Nvidia Monocular Dataset. The results demonstrate a significant leap in efficiency and quality.
- Massive Compression: MoDec-GS achieves an average 70% reduction in model size compared to state-of-the-art methods like 4DGS and SC-GS. On the iPhone dataset, storage requirements were slashed by up to 94% (from ~426MB to ~56MB).
- Superior Rendering Quality: Despite its compact size, the model often improves rendering quality. On the iPhone dataset, it achieved a PSNR gain of +0.7dB over the next best method.
- Real-Time Performance: The framework maintains rendering speeds of ~23-24 FPS, making it suitable for interactive applications on consumer-grade hardware.
Conclusion
MoDec-GS successfully addresses the storage and complexity challenges of dynamic 3D Gaussian Splatting. By decomposing motion and adaptively adjusting temporal intervals, it achieves state-of-the-art performance with a significantly smaller memory footprint. While the model currently faces challenges with ultra-fine details like broom bristles, future work will explore integrating traditional graphics techniques like texture mapping and hierarchical pyramid features to further enhance detail without compromising efficiency.
Summary updated