HEIR: Learning Graph-Based Motion Hierarchies

Cheng Zheng, William Alexander Koch, Baiang Li, Felix Heide

Introduction

Motion in the physical world is rarely a collection of independent, stochastic movements. Instead, it is inherently structured and hierarchical: a finger’s trajectory is defined relative to the hand, which in turn moves relative to the arm. Capturing these "parent-child" relationships is essential for high-fidelity computer vision, robotics, and graphics.

Historically, researchers have approached motion modeling through two primary lenses. The first involves manual templates, such as pre-defined skeletal rigs. While effective for human characters, these templates fail to generalize to arbitrary objects like excavators, folding fabrics, or fluid splashes. The second approach utilizes black-box neural networks to predict motion fields directly. While flexible, these models often lack interpretability and frequently produce "rubbery" or physically implausible distortions because they do not account for underlying structural constraints.

HEIR (Hierarchical Motion Learning) introduces a novel framework designed to automatically discover these hierarchies directly from raw trajectory data, requiring no prior knowledge of an object’s category or shape.

The HEIR Framework

HEIR treats motion elements—such as points in space or 3D Gaussian splats—as nodes within a graph. The framework formulates motion decomposition as a differentiable graph learning problem through three core components:

1. The Learnable Motion Graph (Encoder)

HEIR utilizes a Graph Neural Network (GNN) to analyze the absolute velocities of all points. By employing an attention mechanism, the encoder predicts "edge weights" that represent the probability of one node being the parent of another. This transforms the structural discovery process into a continuous optimization problem.

2. Differentiable Sampling

To convert these probabilities into a concrete, discrete hierarchy (specifically a Directed Acyclic Graph), the authors utilize the Gumbel-Softmax trick. This allows the model to select a specific parent for each node while maintaining end-to-end differentiability, enabling the system to learn the optimal structure through backpropagation.

3. The Kinematic Decoder (Neumann Series)

A key innovation of HEIR is its use of a truncated Neumann series to reconstruct total motion. In a hierarchy, a point's global motion is the sum of its local motion and the motion of all its ancestors. The Neumann series effectively "unrolls" the tree structure, allowing the model to calculate how global motion trickles down to local parts efficiently. To ensure the hierarchy is meaningful, the authors apply an $\ell_1$ penalty to the relative motion. This forces the model to explain as much movement as possible through the hierarchy, leaving only minimal residuals for local adjustments.

Experimental Evaluation

The authors evaluated HEIR across three distinct scales to test its robustness and accuracy:

  • 1D Toy Example: A synthetic dataset of 11 nodes moving in nested sine waves. HEIR successfully recovered the ground-truth hierarchy in 73% of trials, whereas random chance would yield a success rate of effectively zero ($5.0 \times 10^{-7}\%$).
  • 2D Planetary System: A "star-planet-moon" simulation designed to test the model’s ability to handle rotational inheritance and complex orbital paths.
  • D-NeRF Benchmark: A standard for dynamic 3D scene reconstruction. Testing on complex scenes like the Excavator, Hook, and Warrior, HEIR was integrated with 3D Gaussian Splatting to evaluate its performance on thousands of points.

Key Results and Insights

Quantitative Superiority

In the D-NeRF Excavator scene, HEIR significantly outperformed the state-of-the-art SC-GS baseline. It achieved a PSNR of 21.56 (compared to 19.91) and a substantially lower LPIPS (perceptual error) of 0.0383 (compared to 0.065). These metrics indicate that the hierarchical approach leads to much more accurate reconstructions of dynamic scenes.

Structural Rigidity and Interpretability

Qualitatively, HEIR preserves the physical integrity of objects. While baseline methods often cause rigid parts (like a metal excavator arm) to stretch or bend unnaturally, HEIR correctly identifies these parts as rigid units following a parent joint. Furthermore, because HEIR produces an explicit graph, the resulting model is highly interpretable. Users can inspect the discovered hierarchy and even manually manipulate "handles" within the graph to control the animation.

Limitations and Future Directions

Despite its success, HEIR faces several challenges:

  • The Static Problem: The model can only infer relationships from observed motion. If a component (e.g., a shovel on a parked truck) remains stationary during training, its hierarchical position cannot be determined.
  • Single-Parent Constraint: The current model assumes a tree structure where each node has one parent. Future iterations may need to support Directed Acyclic Graphs with multiple parents to model complex constraints like mechanical loops.
  • Long-Range Dependencies: Currently, the model relies on $k$-nearest neighbors for its initial graph. Moving toward global attention mechanisms could help capture correlations between spatially distant but motion-linked parts.

Summary updated

Abstract

Hierarchical structures of motion exist across research fields, including computer vision, graphics, and robotics, where complex dynamics typically arise from coordinated interactions among simpler motion components. Existing methods to model such dynamics typically rely on manually-defined or heuristic hierarchies with fixed motion primitives, limiting their generalizability across different tasks. In this work, we propose a general hierarchical motion modeling method that learns structured, interpretable motion relationships directly from data. Our method represents observed motions using graph-based hierarchies, explicitly decomposing global absolute motions into parent-inherited patterns and local motion residuals. We formulate hierarchy inference as a differentiable graph learning problem, where vertices represent elemental motions and directed edges capture learned parent-child dependencies through graph neural networks. We evaluate our hierarchical reconstruction approach on three examples: 1D translational motion, 2D rotational motion, and dynamic 3D scene deformation via Gaussian splatting. Experimental results show that our method reconstructs the intrinsic motion hierarchy in 1D and 2D cases, and produces more realistic and interpretable deformations compared to the baseline on dynamic 3D Gaussian splatting scenes. By providing an adaptable, data-driven hierarchical modeling paradigm, our method offers a formulation applicable to a broad range of motion-centric tasks.
BibTeX
@inproceedings{zheng2025heir,
  author = {Cheng Zheng and William Alexander Koch and Baiang Li and Felix Heide},
  title = {{HEIR: Learning Graph-Based Motion Hierarchies}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2025},
  url = {https://openreview.net/forum?id=8VkTkb6WL6},
}
Figure