Robots that push a door shut, swing a hanging bag, or flick a drawer closed need to predict how things will move several seconds ahead. They have to make that prediction from cameras alone, and the objects carry unknown masses, springs and friction. Two tools can each solve part of this. Lagrangian neural networks learn a system's kinetic and potential energy and derive motion from them, which keeps long predictions physically sane. Dynamic scene reconstruction methods split a video into a still background and moving pieces. The two have never been joined in a way that works in cluttered, changing scenes. The one method that learns physics-structured dynamics straight from 3D sensing assumes that everything except the robot is frozen.
We propose to learn a Lagrangian model for each moving part of a reconstructed scene and to fit it with a loss computed only on the parts that actually move. The model is trained in simulation over wide randomized distributions of mass, stiffness, damping and appearance, so that a small encoder learns to infer an object's physical parameters from a few seconds of 3D motion rather than averaging over them. The claim is specific. On constrained mechanisms whose energy we do not know in advance (spring-loaded hinges, dampers, cable-hung objects), energy-structured dynamics fitted to reconstructed motion should extrapolate further and transfer to real hardware better than unconstrained force regressors trained on the same tracks. It should also beat end-to-end policies trained under domain randomization, which tend to become overly cautious.
What exists and where the gap is
Lagrangian networks supply the mechanism. Deep Lagrangian Networks learn a positive-definite mass matrix and a scalar potential and obtain the equations of motion by automatic differentiation. When tested at twice the training velocity, black-box networks' errors exploded while DeLaN's stayed low. On the cable-driven Barrett WAM, however, DeLaN lost to an MLP in some regimes because it could not express friction and cable elasticity. That negative result constrains our design: dissipation must be modeled explicitly. OSCAR shows how to do this. It pairs a DeLaN base with a multiplicative residual and a "latent extrinsics" encoder that infers payload and friction from short state histories under simulated dynamics randomization. We reuse that structure, but OSCAR only ever sees joint states and is evaluated only in simulation. Floating-Base Deep Lagrangian Networks extend the idea to quadrupeds on real hardware, again from proprioception alone.
The closest bridge to perception is Hamiltonian Dynamics Learning from Point Cloud Observations. It trains an energy-structured neural ODE from LiDAR with an observation-space cycle-consistency loss: integrate forward, transform the current point cloud, and register it against the next one. It converged in 400 iterations on a real Jackal. That loss only makes sense if every change in the cloud comes from the robot's own motion, and its mass matrix is constant. Both assumptions fail for articulated objects in a scene that also contains moving people and robot arms. Lifting those two assumptions is the core technical step of this proposal.
Dynamic reconstruction supplies the perception. Split4D represents a scene as 3D Gaussians (small colored blobs rendered by splatting) that carry velocities and identity features, and separates moving instances without 2D video tracking (mIoU 0.893 vs 0.592 for SA4D on Multi-Human). Its motion model is purely kinematic. R5DGS binds Gaussians into rigid groups for faster extrapolation and reports a photometric cost in deformable regions. Its extrapolation has no energy, mass or torque. Neural Gaussian Force Fields predict per-object forces with a neural operator, train on 640k simulated videos and transfer zero-shot to real footage. Its forces are regressed, not derived from an energy, and it targets falling and colliding objects rather than constrained mechanisms driven by a robot. NGFF is the closest prior work. Our difference is that motion comes from learned energies of discovered joint coordinates, and physical parameters are inferred per object under randomization instead of being absorbed into a single regressor.
Core Claim
Start from a Split4D-style decomposition and fit joint coordinates to the relative motion of moving Gaussian groups. On those coordinates, a Lagrangian model with a dissipation residual and a latent-parameter encoder, trained across domain-randomized simulation with a cycle-consistency loss restricted to moving groups, predicts multi-second motion of real articulated objects more accurately than unconstrained dynamics on the same coordinates. The gain should be largest at long horizons and outside the training velocity range. It should also make one-shot dynamic manipulation succeed more often than a domain-randomized end-to-end policy.
Research Program
The pipeline has three learned stages and one classical stage.
The first stage is perception. A multi-view or RGB-D 4D Gaussian reconstruction with Split4D-style identity features separates static background, the robot (masked using its known kinematics) and moving object groups. We stay with multi-view input because monocular dynamic reconstruction remains open, as the file notes for all current 4D methods.
The second stage is classical. For each pair of moving groups we fit a screw axis to their relative pose track, which yields revolute or prismatic joint coordinates $q$. A free group keeps a 6D pose.
The third stage is the dynamics model. For the coordinates $q$ we learn a mass matrix $M_\theta(q)$ through a Cholesky factor as in DeLaN, a potential $V_\theta(q)$ that captures springs and gravity, and a dissipation term $D_\theta(q,\dot q)$ with OSCAR-style multiplicative correction. Each of these is conditioned on a latent $z$ from an encoder that reads a short history of reconstructed coordinates and, when available, robot wrist torques. Motion follows
$$M_\theta(q,z)\ddot q + C_\theta(q,\dot q,z)\dot q + \nabla_q V_\theta(q,z) = \tau - D_\theta(q,\dot q,z),$$
where $C_\theta$ is computed from $M_\theta$ via Christoffel symbols and $\tau$ is the generalized force from robot contact projected through the contact Jacobian.
Training uses a loss defined in observation space, and only on moving content. We integrate $q$ forward $k$ steps, move each group's Gaussians rigidly by the implied poses, and penalize the distance between the moved Gaussian centers of moving groups and those reconstructed at $t+k$. Background and robot Gaussians are excluded, which is what removes the static-scene assumption of the point-cloud Hamiltonian work.
Simulation data come from SAPIEN, the engine underlying ManiSkill2. We procedurally generate cabinets, hinged lids, drawers and pendulum-hung objects. Randomization covers link masses and inertias, spring stiffness and rest angle, damping, Coulomb friction, and the visual factors in the spirit of (CAD)²RL: textures, lighting and camera placement. We also add distractors, people and second robots moving through view, so that the decomposition is stress-tested.
At deployment the learned model drives a sampling-based model-predictive controller. The test tasks require calibrated dynamics: give a spring-loaded door a single push so that it stops at a target angle, swing a hung object to a target height, or close a damped drawer with one impulse.
Baselines
The naive reference is a constant-velocity extrapolation of the reconstructed coordinates, which is effectively Split4D's kinematic motion model. Every learned model must beat it at all horizons.
The strongest dynamics competitor is an NGFF-style unconstrained force regressor. It is a network that maps $(q,\dot q,\tau,z)$ to $\ddot q$. It receives the same decomposition, coordinates, latent encoder, randomized data, number of gradient steps and hyperparameter search budget as our model. Only the Lagrangian structure is removed. This is the fairest test of whether energy structure matters once perception is held fixed. We do not use NGFF's full pipeline as the head-to-head comparison because its dataset targets collisions rather than robot-actuated mechanisms. We do report it on its own GSCollision domain only as a sanity check. The file gives no reported RMSE or final-position-error values for NGFF ("no reported result found on Lacuna for NGFF trajectory error"), so we will rerun it rather than quote a number.
The second strong competitor is classical system identification. A rigid-body simulator is given the fitted joint types, and its masses, spring constants and damping are fitted by gradient-free optimization to the same video-derived tracks.
The policy-level competitor is an end-to-end point-cloud policy trained under the same domain randomization, with a privileged critic as in Asymmetric Actor Critic. That method reports 5/5 real-world success on picking, pushing and moving blocks. ManiSkill2 reports a sim-trained PickCube policy reaching 91% in simulation and 60% on hardware without fine-tuning. Neither number comes from our tasks, and we do not claim it as a target. For dynamic articulated manipulation from vision, no reported result was found on Lacuna, so all baselines will be run in-house under matched simulator samples.
Three ablations isolate the key ingredients. The first removes the moving-only mask and applies the cycle loss to the whole scene, as in the Hamiltonian point-cloud method. The second removes the latent encoder and trains on one fixed parameter setting, which tests randomization-with-inference against plain randomization. The third removes the dissipation residual, which tests the DeLaN WAM failure mode.
Beating the strongest baseline means three things. Our model must show lower angle error at a 3-second horizon on real hardware. It must show lower error on velocities outside the training range. And it must reach higher single-shot task success with overlapping confidence intervals excluded across at least five physical objects.
Why not just…?
Why not just regress forces with an unconstrained network on the reconstructed tracks? This is NGFF's design applied to our coordinates. DeLaN's 2× velocity experiment suggests that unconstrained models fail precisely in extrapolation, and single-push tasks reach peak velocities that slow demonstrations never cover. The matched-budget comparison above settles this directly. If the unconstrained model ties ours on out-of-range velocities and at long horizons, the energy structure is not earning its cost, and we will report that.
Why not just fit a rigid-body simulator's parameters to the tracks? This is a real threat. When the mechanism is exactly expressible by the simulator, such as an ideal hinge with linear damping, classical identification has the right structure and few parameters, and it may match or beat us. Our claim is restricted to mechanisms whose potential and dissipation are not in the simulator's vocabulary, such as nonlinear gas struts, soft-close dampers and cable suspensions. We will therefore split the real test set into "engine-expressible" and "non-expressible" objects and report results on each. If identification wins on both splits, the proposal's thesis fails.
Why not just train an end-to-end point-cloud policy under domain randomization? ManiSkill2 found point clouds outperform RGB-D for complex manipulation, so this is the natural visual baseline. But a policy that must succeed across every sampled stiffness and mass at once tends toward cautious, repeated pushes. That is the over-conservatism problem the file documents for broad randomization. Our model instead infers the parameters and then plans. We measure single-shot success and the number of contacts per episode. If the end-to-end policy matches our single-shot success, the dynamics model is unnecessary for control, and its value reduces to prediction alone.
Proposed Experiments
Simulation evaluation uses held-out procedural mechanisms whose parameters lie inside and outside the randomization ranges. Each scene also contains moving distractors. We measure joint-angle error at 0.5, 1 and 3 seconds, energy drift on undamped test cases, and correlation of the latent $z$ with the true stiffness and mass.
As a calibration step, we first reproduce DeLaN's simulated 2-DOF and 7-DOF results on true states. This separates errors in the Lagrangian component from errors in perception.
On hardware, a Franka arm with a calibrated multi-camera rig manipulates real cabinets, struts and hung objects while people walk through the scene. We report the same prediction metrics against motion-capture ground truth and success on the three single-shot tasks. The Dynamic Indoor Scene and Multi-Human benchmarks serve only to confirm that our decomposition stays close to Split4D's reported segmentation quality.
The decisive checks are three. The first compares Lagrangian and matched unconstrained models on real long-horizon, out-of-range-velocity prediction. The second compares against classical identification on the non-expressible split. The third tests whether the moving-only loss holds its accuracy when distractors are added, while the whole-scene loss degrades.
References
- Deep Lagrangian Networks: Using Physics as Model Prior for Deep Learning
- OSCAR: Data-Driven Operational Space Control for Adaptive and Robust Robot Manipulation
- Floating-Base Deep Lagrangian Networks
- Hamiltonian Dynamics Learning from Point Cloud Observations for Nonholonomic Mobile Robot Control
- Split4D: Decomposed 4D Scene Reconstruction Without Video Segmentation
- R5DGS: Semantic-Aware 4D Gaussian Splatting with Rigid Body Constraints for Efficient Dynamic Scene Reconstruction
- Learning Physics-Grounded 4D Dynamics with Neural Gaussian Force Fields
- ManiSkill2: A Unified Benchmark for Generalizable Manipulation Skills
- CAD2RL: Real Single-Image Flight Without a Single Real Image
- Asymmetric Actor Critic for Image-Based Robot Learning