Predicting how a physical system—like a swinging pendulum or a moving drone—will behave is a cornerstone of robotics. While we have the mathematical frameworks (Lagrangian and Hamiltonian mechanics) to describe these systems, applying that math directly to high-dimensional visual data like raw video frames is notoriously difficult.
Most current approaches fall into two traps. First, they often depend heavily on supervision, requiring "ground truth" coordinate data (e.g., the exact angle of a joint) which is rarely available in real-world video. Second, unsupervised models often suffer from the "black box" problem: they learn latent representations that are just a mess of numbers. While these might help with short-term prediction, they offer no physical insight and cannot be used to design traditional energy-based controllers.
To address this, researchers have developed a neural network that learns the underlying laws of physics (Lagrangian dynamics) directly from raw video frames without any manual labeling of coordinates. By combining a "coordinate-aware" Variational Autoencoder (VAE) with a physics-constrained latent space, the model predicts long-term motion accurately and learns interpretable energy functions that can be used to design controllers for complex mechanical systems.
Methodology
The authors introduce a framework consisting of two main components trained end-to-end:
1. Coordinate-Aware VAE
Standard VAEs struggle to map images to meaningful physical coordinates like angles. To fix this, the authors designed a "coordinate-aware" architecture:
- Spatial Transformer Networks (STN): Instead of relying on a black-box Convolutional Neural Network (CNN), the model uses an STN to shift its "attention window" based on learned translational coordinates.
- von Mises Distribution: To handle rotations, the model uses the von Mises distribution (a "circular" version of a Gaussian) to represent angles. This ensures the latent space respects the periodic nature of rotation (i.e., $0$ and $2\pi$ represent the same position).
2. Latent Lagrangian Dynamics
Inside the latent space, the model doesn't just learn an arbitrary transition function. It is forced to obey the Euler-Lagrange equations. The network learns three specific sub-functions parameterized as neural networks:
- $M(q)$: The mass matrix (kinetic energy).
- $V(q)$: The potential energy.
- $g(q)$: The control input matrix.
By integrating these functions using a Neural ODE solver, the model ensures that the predicted motion is physically consistent, such as conserving energy when no control is applied.
Datasets and Evaluation
The model was evaluated on three classic physical systems simulated via OpenAI Gym:
- Pendulum: A single rotating arm.
- CartPole: A pole balanced on a moving cart, requiring both translation and rotation tracking.
- Acrobot: A two-link robot arm representing a chaotic system.
The training data consisted solely of image sequences and the control inputs applied during those sequences. Crucially, no coordinate data (angles or positions) were provided to the model during training.
Key Insights and Results
- Interpretability Enables Control: Because the model learns a literal "potential energy" function from pixels, the authors could use classical control theory (Energy Shaping) to move the systems to target positions. For example, they could command the CartPole to "go to the state with maximum potential energy" to make it stand upright.
- Physics as a Regularizer: Enforcing Lagrangian structure acts as a powerful inductive bias. Standard Multi-Layer Perceptrons (MLPs) tend to "drift" or gain/lose energy over long-term predictions, whereas this model maintains physical realism over much longer horizons.
- The Geometry of the Encoder Matters: The paper highlights that black-box encoders fail at extracting coordinates from images of multi-body systems. By explicitly accounting for geometry—such as shifting the attention window for the pole based on the cart's position—the model successfully disentangles complex motions.
- Superior Prediction and Zero-Shot Control: The proposed model significantly outperformed baselines like Hamiltonian Generative Networks (HGN) and standard MLPs in long-term pixel-wise prediction. Furthermore, the researchers successfully implemented PD-controllers with energy compensation using the learned parameters, swinging up and balancing the Pendulum and CartPole using only the model learned from raw images.
Limitations and Future Work
While groundbreaking, the current framework has a few limitations:
- Planar Constraint: The current work is limited to rigid bodies moving in a 2D plane. Extending this to 3D environments with occlusions would be a significant next step.
- Segmentation Requirement: The model currently assumes that the number of rigid bodies is known and that a basic segmentation of the objects in the image is provided.
- Complexity Scaling: While it handles the Acrobot, scaling this unsupervised coordinate discovery to systems with many more degrees of freedom (like a humanoid robot) remains an open challenge.
Summary updated