TL;DR
HEIR (Hierarchical Motion Learning) is a novel data-driven framework that automatically learns structured, interpretable motion hierarchies directly from raw trajectory data. By formulating hierarchy inference as a differentiable graph learning problem, it successfully decomposes complex global dynamics into parent-inherited motions and local residuals, significantly improving the fidelity and control of 3D scene deformations.
What Problem Does This Paper Address?
Complex motion—whether it's a human body moving, a robotic arm flexing, or an excavator digging—is inherently hierarchical. Sub-components move relative to their parent structures.
Current methods to model these dynamics typically fall into two flawed categories:
- Explicit/Manual Hierarchies: Relying on hand-crafted kinematic templates (like skeletons). These are rigid, domain-specific, and fail to generalize to novel or unstructured objects.
- Implicit Neural Representations: Learning motion end-to-end without explicit structure. While flexible, these methods lack interpretability and often suffer from entangled deformation artifacts (e.g., parts of an object stretching or bending unrealistically during editing).
HEIR bridges this gap by discovering the underlying motion hierarchy automatically from data without assuming any prior domain knowledge or pre-defined structural templates.
Methods
HEIR models motion using a learnable Directed Acyclic Graph (DAG), where vertices represent elemental motion components (like points or Gaussian splats) and directed edges define parent-child dependencies.
The architecture consists of an Encoder-Decoder structure optimized end-to-end:
- Sparse Message Passing (Encoder): Operating on a spatial proximity graph, a Graph Attention Network (GAT) predicts edge weights that represent the probability of any node being another node's parent.
- Differentiable Sampling: To maintain a discrete tree structure while allowing backpropagation, the framework utilizes the Gumbel-Softmax trick to sample a valid hierarchy matrix ($H$) during training.
- Absolute Motion Reconstruction (Decoder): The absolute velocity of any element is decomposed as $\Delta^t = H\Delta^t + \delta^t$, where $\delta^t$ is the local residual velocity. The decoder reconstructs global motion by accumulating these relative velocities along the learned hierarchy using a truncated Neumann series.
- Rotation Extension: The authors also extend the encoder to predict relative velocities in polar coordinates, allowing the framework to naturally inherit rotational mechanics.
Datasets
The authors evaluated HEIR across three distinct setups:
- 1D Hierarchical Motion Toy Benchmark: A synthetic dataset of 11 nodes moving via nested, multi-frequency sine wave trajectories to test structural expressiveness.
- Planetary System: A synthetic 2D dataset modeling rotational hierarchies (stars, planets, and moons) to evaluate rotation inheritance.
- D-NeRF Dataset: A standard benchmark for dynamic 3D scene reconstruction featuring complex objects like an Excavator, Hook, Jumpingjacks, and a Warrior.
Key Insights and Ideas
- The Power of the Residual Regularizer: A critical component of the training objective is an $\ell_1$ regularization penalty on the magnitude of the relative velocity ($\delta^t$). Without this, the model would take a trivial shortcut: setting the relative velocity equal to the absolute velocity and bypassing the hierarchy entirely (creating a flat "star" topology). Forcing the local residuals to be small compels the network to explain motion through parent inheritance.
- Neumann Series for Stability: Representing the hierarchy as a truncated Neumann series ensures that the absolute motion extraction remains stable and well-defined during training, even if the probabilistic sampling occasionally introduces temporary cycles before the graph converges into a strict DAG.
Key Results
HEIR demonstrated exceptional accuracy in discovering latent structures and generating high-fidelity deformations:
- Structure Discovery: In the 1D toy benchmark, HEIR recovered the exact underlying motion hierarchy in 73% of cases out of over $10^{10}$ potential candidate graph configurations. In the noise-free planetary dataset, it achieved a 100% reconstruction rate.
- 3D Scene Deformation: When integrated with 3D Gaussian Splatting on the D-NeRF dataset, HEIR consistently outperformed the state-of-the-art SC-GS baseline across all perceptual and structural metrics.
| Scene | Metric | SC-GS [9] | Ours (HEIR) |
|---|---|---|---|
| Excavator | PSNR ↑ <br> LPIPS ↓ | 19.91 <br> 0.0650 | 21.56 <br> 0.0383 |
| Hook | PSNR ↑ <br> LPIPS ↓ | 15.70 <br> 0.0954 | 18.30 <br> 0.0617 |
Qualitatively, when a user manipulates a "handle" on an object, HEIR preserves structural rigidity and natural limb alignment, whereas baseline methods introduce unnatural bending and geometric distortions.
Limitations and Future Work
Despite its strong performance, HEIR has a few notable limitations:
- Dependence on Observable Motion: The model can only infer relationships from active motion. If a specific part of an object remains static throughout the training sequence (e.g., an excavator's shovel that doesn't move), HEIR cannot discover its latent structural boundaries or semantics.
- Single-Parent Assumption: The current formulation assumes each motion element has exactly one parent. This tree-like structure restricts its expressiveness in complex physical systems featuring overlapping or multi-source motion influences (like cloth dynamics or fluid-structure interactions).
Future work aims to address these long-range dependencies by exploring global-connectivity variants beyond standard $k$-NN proximity graphs.
Summary updated