Motion Decoupled 3D Gaussian Splatting for Dynamic Object Representation

Xiao Hu, Libo Long, Jochen LangAAAI 2025 · DOI

Abstract

M5D-GS is a novel 3D Gaussian Splatting framework that handles dynamic objects undergoing large-scale movements by explicitly decoupling global rigid motion from local deformations. By separating "where the object is" from "how it is changing shape," it achieves state-of-the-art accuracy on complex scenes while maintaining real-time rendering speeds of over 130 FPS.

Introduction

Dynamic 3D scene reconstruction—creating a digital twin of a moving object from a single camera—is a critical challenge for AR/VR and autonomous driving. Current state-of-the-art methods typically utilize a canonical space (a static template of the object) and a deformation field (a mathematical map that tells the template how to move at any given time).

However, this approach often fails when objects undergo significant translation or rotation. Large motions create two primary obstacles:

  1. Ambiguity: The system struggles to distinguish whether a change in appearance is caused by global movement or local deformation.
  2. Initialization Failure: If the initial predicted position is too far from the actual location, the lack of spatial overlap between the model and the ground truth prevents effective optimization, causing the system to get stuck in local minima.
M5D-GS handles a variety of dynamic objects undergoing complex motions.

Methodology: Motion Decoupling

The authors introduce Motion Decoupled Dynamic 3D Gaussian Splatting (M5D-GS). The core innovation is the explicit separation of global rigid motion from local non-rigid changes. The architecture consists of three main trainable components:

  • Canonical 3D Gaussians: A set of explicit 3D points representing the object’s base structure.
  • Object-Level Motion MLP ($\Theta_{motion}$): A neural network that predicts a single rotation (quaternion) and translation vector for the entire object at any timestamp $t$, handling large-scale rigid movements.
  • Per-Gaussian Deformation MLP ($\Theta_{deform}$): A separate network that predicts local offsets for individual Gaussian points, capturing subtle non-rigid changes like limb articulation.

Coarse-to-Fine Matching

To address initialization failures, M5D-GS employs a "warmup" strategy. During early training, the 3D Gaussians are artificially inflated. This increased scale ensures sufficient overlap with the ground truth even when the initial motion prediction is inaccurate, providing a valid gradient signal. As training stabilizes, the Gaussians are gradually restored to their intended size.

Datasets and Evaluation

Standard benchmarks like D-NeRF often lack the large-scale motion necessary to test robust reconstruction. To address this, the authors developed a more challenging benchmark consisting of 10 diverse scenarios, including:

  • Augmented D-NeRF: Existing datasets modified to include large rigid motions.
  • Synthetic and Real-World Scenes: New scenarios featuring fast-moving objects in both simulated and practical environments.
Comparison between standard motion (top) and the challenging large-scale motion (bottom) addressed by M5D-GS.

Results and Discussion

M5D-GS demonstrates significant improvements in both robustness and accuracy compared to existing methods like D3D-GS and 4D-GS.

  • Accuracy: On the "Jumping Jacks + Motion" scene, M5D-GS achieved a PSNR of 35.86, whereas competing methods often failed to reconstruct the object correctly under high translation.
  • Efficiency: The model maintains a rendering speed of 135.15 FPS. While slightly slower than some baselines due to the dual-MLP architecture, it remains well within the requirements for real-time applications.
  • Theoretical Grounding: The paper introduces the Overlap Coefficient (OC), providing statistical proof that the probability of successful optimization is directly tied to the ratio of Gaussian scale to translation error.

Conclusion

By decoupling global motion from local deformation, M5D-GS effectively addresses the limitations of current dynamic 3D reconstruction techniques. While currently limited to single-object scenes and monocular views, it sets a new standard for representing objects undergoing large-scale movements. Future work will focus on multi-object scenarios and incorporating spatial-temporal constraints to further reduce overfitting.

Summary updated

BibTeX
@inproceedings{hu2025motion,
  author = {Xiao Hu and Libo Long and Jochen Lang},
  title = {{Motion Decoupled 3D Gaussian Splatting for Dynamic Object Representation}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i4.32373},
}
Figure