HiMoR: Monocular Deformable Gaussian Reconstruction with Hierarchical Motion Representation

Yiming Liang, Tianhan Xu, Yuta KikuchiCVPR 2025 · DOI

Introduction

Reconstructing a dynamic 3D scene from a single moving camera—monocular video—remains one of the most challenging problems in computer vision. This "highly ill-posed" task requires the system to disentangle complex object motion from depth ambiguity without the benefit of multiple simultaneous viewpoints.

Current state-of-the-art approaches often leverage 3D Gaussian Splatting (3DGS) for its efficiency and rendering quality. However, they typically struggle with a trade-off between expressiveness and stability. Methods using a limited set of global motion bases often fail to capture fine-grained details, such as finger movements. Conversely, methods that allow for high-degree-of-freedom independent motion nodes frequently over-fit to the training camera's perspective, leading to "geometry collapse" and visual artifacts when viewed from novel angles.

Hierarchical Motion Representation (HiMoR)

To address these limitations, the authors propose HiMoR, a framework that structures motion as a coarse-to-fine tree graph. This approach is rooted in the observation that physical motion is naturally hierarchical: an arm's movement dictates the general position of the hand, which in turn provides the local coordinate system for the fingers.

Overview of the HiMoR framework, illustrating the hierarchical tree structure, shared motion bases, and Gaussian deformation via KNN interpolation.

Core Components

  1. Hierarchical Tree Structure: HiMoR models motion relative to parent nodes. The root node is fixed at the world origin. Shallow nodes capture coarse, global movements, while deeper nodes represent fine-grained local deformations.
  2. Shared Motion Bases: To prevent chaotic or unrealistic movements, child nodes under the same parent share a set of $SE(3)$ motion bases. The specific motion of a node is determined by a weighted combination of these bases, effectively imposing a low-rank constraint that maintains physical plausibility.
  3. Gaussian Deformation: The 3D Gaussians representing the scene's appearance determine their world-space position by interpolating from the $K$-nearest neighbor (KNN) leaf nodes of the motion tree. This process uses dual quaternion skinning to ensure smooth transitions across the volume.
  4. Adaptive Node Densification: The hierarchy is not static. The model begins with a simple structure and automatically adds nodes in regions where motion is complex—specifically where a Gaussian's trajectory significantly deviates from its neighbors.
  5. Hierarchical Rigidity Loss: To ensure structural integrity, the authors apply varying levels of rigidity constraints. Shallower nodes are subject to stronger constraints to maintain global shape, while deeper nodes are granted more flexibility to capture subtle details.

Evaluation and the "PSNR Trap"

The researchers evaluated HiMoR on the iPhone Dataset (14 real-world scenes with complex occlusions) and the Nvidia Dataset (monocular training for novel view synthesis).

A key insight from this work is the critique of standard metrics like Peak Signal-to-Noise Ratio (PSNR). In monocular dynamic reconstruction, a perfectly rendered object that is slightly misaligned due to minor camera calibration errors will receive a poor PSNR score. This "PSNR Trap" fails to account for the actual visual quality and temporal consistency of the reconstruction.

Qualitative comparison showing artifacts in the SoM baseline. Note the "broken" or "transparent" appearance of the hand and apple, despite a higher PSNR than some other frames.

To better reflect human perception, the authors emphasize perceptual metrics like CLIP-I (image similarity) and CLIP-T (temporal consistency) to better reflect the actual quality of the 3D reconstruction.

Key Results

HiMoR demonstrates significant improvements over existing NeRF-based and 3DGS-based methods. On the iPhone Dataset, it achieved a CLIP-I score of 0.8907 and a CLIP-T score of 0.9750, outperforming competitors like SoM and HyperNeRF. Qualitatively, HiMoR avoids the "broken" or "transparent" artifacts common in other methods when rendering fast-moving objects or complex articulations from novel perspectives.

Limitations and Future Work

The primary constraint of HiMoR is its reliance on a Canonical Frame. Like many 3DGS methods, it struggles to represent scene elements that are not present in the initial frame, such as an object being pulled from a pocket. Future research will focus on developing adaptive canonical spaces and separate hierarchical branches to handle the appearance of new objects dynamically.

Summary updated

BibTeX
@inproceedings{liang2025himor,
  author = {Yiming Liang and Tianhan Xu and Yuta Kikuchi},
  title = {{HiMoR: Monocular Deformable Gaussian Reconstruction with Hierarchical Motion Representation}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2025},
  doi = {10.1109/cvpr52734.2025.00091},
}
Figure