HEIR: Learning Graph-Based Motion Hierarchies

Cheng Zheng, William Koch, Baiang Li, Felix Heide

TL;DR

HEIR (Hierarchical Motion Learning) is a novel data-driven framework that automatically learns structured, interpretable motion hierarchies directly from raw trajectory data. By formulating hierarchy inference as a differentiable graph learning problem, it successfully decomposes complex global dynamics into parent-inherited motions and local residuals, significantly improving the fidelity and control of 3D scene deformations.

What Problem Does This Paper Address?

Complex motion—whether it's a human body moving, a robotic arm flexing, or an excavator digging—is inherently hierarchical. Sub-components move relative to their parent structures.

Current methods to model these dynamics typically fall into two flawed categories:

  1. Explicit/Manual Hierarchies: Relying on hand-crafted kinematic templates (like skeletons). These are rigid, domain-specific, and fail to generalize to novel or unstructured objects.
  2. Implicit Neural Representations: Learning motion end-to-end without explicit structure. While flexible, these methods lack interpretability and often suffer from entangled deformation artifacts (e.g., parts of an object stretching or bending unrealistically during editing).

HEIR bridges this gap by discovering the underlying motion hierarchy automatically from data without assuming any prior domain knowledge or pre-defined structural templates.

Methods

HEIR models motion using a learnable Directed Acyclic Graph (DAG), where vertices represent elemental motion components (like points or Gaussian splats) and directed edges define parent-child dependencies.

Overview of the HEIR framework

The architecture consists of an Encoder-Decoder structure optimized end-to-end:

  • Sparse Message Passing (Encoder): Operating on a spatial proximity graph, a Graph Attention Network (GAT) predicts edge weights that represent the probability of any node being another node's parent.
  • Differentiable Sampling: To maintain a discrete tree structure while allowing backpropagation, the framework utilizes the Gumbel-Softmax trick to sample a valid hierarchy matrix ($H$) during training.
  • Absolute Motion Reconstruction (Decoder): The absolute velocity of any element is decomposed as $\Delta^t = H\Delta^t + \delta^t$, where $\delta^t$ is the local residual velocity. The decoder reconstructs global motion by accumulating these relative velocities along the learned hierarchy using a truncated Neumann series.
  • Rotation Extension: The authors also extend the encoder to predict relative velocities in polar coordinates, allowing the framework to naturally inherit rotational mechanics.

Datasets

The authors evaluated HEIR across three distinct setups:

  1. 1D Hierarchical Motion Toy Benchmark: A synthetic dataset of 11 nodes moving via nested, multi-frequency sine wave trajectories to test structural expressiveness.
  2. Planetary System: A synthetic 2D dataset modeling rotational hierarchies (stars, planets, and moons) to evaluate rotation inheritance.
  3. D-NeRF Dataset: A standard benchmark for dynamic 3D scene reconstruction featuring complex objects like an Excavator, Hook, Jumpingjacks, and a Warrior.

Key Insights and Ideas

  • The Power of the Residual Regularizer: A critical component of the training objective is an $\ell_1$ regularization penalty on the magnitude of the relative velocity ($\delta^t$). Without this, the model would take a trivial shortcut: setting the relative velocity equal to the absolute velocity and bypassing the hierarchy entirely (creating a flat "star" topology). Forcing the local residuals to be small compels the network to explain motion through parent inheritance.
  • Neumann Series for Stability: Representing the hierarchy as a truncated Neumann series ensures that the absolute motion extraction remains stable and well-defined during training, even if the probabilistic sampling occasionally introduces temporary cycles before the graph converges into a strict DAG.

Key Results

HEIR demonstrated exceptional accuracy in discovering latent structures and generating high-fidelity deformations:

  • Structure Discovery: In the 1D toy benchmark, HEIR recovered the exact underlying motion hierarchy in 73% of cases out of over $10^{10}$ potential candidate graph configurations. In the noise-free planetary dataset, it achieved a 100% reconstruction rate.
  • 3D Scene Deformation: When integrated with 3D Gaussian Splatting on the D-NeRF dataset, HEIR consistently outperformed the state-of-the-art SC-GS baseline across all perceptual and structural metrics.
Scene Metric SC-GS [9] Ours (HEIR)
Excavator PSNR ↑ <br> LPIPS ↓ 19.91 <br> 0.0650 21.56 <br> 0.0383
Hook PSNR ↑ <br> LPIPS ↓ 15.70 <br> 0.0954 18.30 <br> 0.0617

Qualitatively, when a user manipulates a "handle" on an object, HEIR preserves structural rigidity and natural limb alignment, whereas baseline methods introduce unnatural bending and geometric distortions.

Qualitative evaluation on D-NeRF scenes

Limitations and Future Work

Despite its strong performance, HEIR has a few notable limitations:

  • Dependence on Observable Motion: The model can only infer relationships from active motion. If a specific part of an object remains static throughout the training sequence (e.g., an excavator's shovel that doesn't move), HEIR cannot discover its latent structural boundaries or semantics.
  • Single-Parent Assumption: The current formulation assumes each motion element has exactly one parent. This tree-like structure restricts its expressiveness in complex physical systems featuring overlapping or multi-source motion influences (like cloth dynamics or fluid-structure interactions).

Future work aims to address these long-range dependencies by exploring global-connectivity variants beyond standard $k$-NN proximity graphs.

Summary updated

Abstract

Hierarchical structures of motion exist across research fields, including computer vision, graphics, and robotics, where complex dynamics typically arise from coordinated interactions among simpler motion components. Existing methods to model such dynamics typically rely on manually-defined or heuristic hierarchies with fixed motion primitives, limiting their generalizability across different tasks. In this work, we propose a general hierarchical motion modeling method that learns structured, interpretable motion relationships directly from data. Our method represents observed motions using graph-based hierarchies, explicitly decomposing global absolute motions into parent-inherited patterns and local motion residuals. We formulate hierarchy inference as a differentiable graph learning problem, where vertices represent elemental motions and directed edges capture learned parent-child dependencies through graph neural networks. We evaluate our hierarchical reconstruction approach on three examples: 1D translational motion, 2D rotational motion, and dynamic 3D scene deformation via Gaussian splatting. Experimental results show that our method reconstructs the intrinsic motion hierarchy in 1D and 2D cases, and produces more realistic and interpretable deformations compared to the baseline on dynamic 3D Gaussian splatting scenes. By providing an adaptable, data-driven hierarchical modeling paradigm, our method offers a formulation applicable to a broad range of motion-centric tasks. Project Page: https://light.princeton.edu/HEIR/
BibTeX
@inproceedings{zheng2025heir,
  author = {Cheng Zheng and William Koch and Baiang Li and Felix Heide},
  title = {{HEIR: Learning Graph-Based Motion Hierarchies}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2025},
  url = {https://openreview.net/forum?id=8VkTkb6WL6},
}
Figure