When we watch a plastic ball and a steel ball drop from a tower, they look identical mid-air. But the moment they strike the ground, their trajectories diverge completely: one bounces high, while the other thuds and stops. To a machine learning model observing this scene, predicting the next frame is impossible based on visual appearance alone. The model lacks crucial information about the objects' intrinsic properties—their elasticity and mass.
Traditionally, roboticists and physicists solved this by manually measuring and plugging these constants into explicit physics engines. However, a powerful alternative paradigm has emerged: unsupervised latent physical representation. Instead of explicitly programming physics parameters, we can force a neural network to predict the future. To minimize its prediction error, the network is driven to automatically discover, isolate, and encode unobservable physical properties into a continuous latent space.
The Predictive Bottleneck as a Physics Engine
Why does a network trained purely on video or state prediction end up learning physics? The answer lies in the predictive bottleneck. If a model is tasked with forecasting long-term trajectories, simple linear extrapolation of velocity quickly fails when objects interact.
David Zheng et al. demonstrated this mechanism using Perception-Prediction Networks. By training a model to predict future motion from unsupervised observations, the network naturally learns to encode intrinsic traits like mass or charge into latent vectors. Because accurate forecasting over time requires accounting for underlying physical constants, the network uses its latent space to represent these properties to minimize prediction error.
Visual/State History ───► [ Encoder ] ───► Latent Physics Code (Mass, Friction, Stiffness)
│
▼
Future State Prediction ◄── [ Dynamics Model ] ◄─┘
This concept scales from simple particle interactions to complex macroscopic scenes. For instance, Aaron Courville et al. trained a model to predict how a tower of blocks would collapse. By forcing the network to minimize pixel-wise prediction errors of future visual states, the system internally developed concepts of gravity and structural support without any explicit labels, allowing it to generalize to novel block configurations.
When observed movements deviate from what visual velocity suggests, the model must attribute these discrepancies to hidden forces. As Qu Tang et al. showed, treating intrinsic physical properties as latent variables required for long-term trajectory prediction allows models to discover concepts like mass and charge. To maintain accuracy over extended horizons, the model internalizes these underlying forces to explain complex accelerations that visual features alone cannot account for.
Learning Physics Through Interaction
While passive observation can reveal properties like gravity or mass during collisions, many physical attributes remain hidden until an agent actively interacts with the environment. For a robot manipulating deformable or granular materials, properties like stiffness, friction, and granularity must be felt or provoked.
Active exploration acts as a probe to populate the latent physics space. Yen-Ling Tai et al. developed a food-scooping framework where the robot interacts with materials (e.g., stirring) to build an implicit representation of material dynamics. Rather than explicitly modeling complex parameters like material friction or weight, the learned latent space naturally clusters items with similar physical behaviors, enabling the robot to generalize its manipulation strategy to entirely novel food items.
Similarly, Alberta Longhini showed that a robot can learn the elastic properties of deformable objects, like cloth, through self-supervised exploratory pulling interactions. By optimizing the accuracy of future graph-dynamics predictions, the system discovers a latent representation that correlates strongly with ground-truth stiffness without requiring manual material labels.
When vision is occluded, combining interaction history with multi-modal sensing makes these latent representations even more robust:
- Dense Packing: Bo Ai et al. demonstrated that latent representations learned from a brief history of visuo-tactile interactions automatically capture object mass and stiffness. These emergent embeddings allow dynamics models to adapt to the mechanical behaviors of different objects in real-time, outperforming traditional physics simulators in tight, cluttered spaces.
- Deformable Object Tracking: Youngsun Wi et al. showed that embedding contact locations and material stiffness into a latent space reduces geometric reconstruction errors by roughly 27% compared to models relying on explicit, privileged contact information.
Granular and Morphological Scaling
The strategy of learning implicit physics codes extends beyond global object properties; it can also be embedded directly into the spatial primitives of a scene or the structure of a robot itself.
In 3D scene representation, recent methods combine differentiable rendering with latent dynamics. Jinxi Li et al. assigned latent physics codes to individual 3D Gaussian primitives. By optimizing these codes alongside a velocity module to predict future video frames, the model implicitly captures localized physical attributes like mass and friction across diverse material types without manual segmentation masks. Mikel Zhobro et al. similarly showed that distilling particle features into a latent representation yields highly stable 3D simulations directly from RGB videos, outperforming standard 2D video generation methods.
[ 3D Gaussian Primitives ] ──► [ Distilled Latent Physics Codes ] ──► Stable 3D Simulation
Finally, this principle applies to the internal dynamics of robots. Instead of training a unique policy for every robot design, Tao Chen et al. introduced Hardware Conditioned Policies (HCP). By optimizing latent embeddings via backpropagation during multi-robot training, the policy autonomously encodes unobservable internal physical properties—such as joint friction, damping, and mass distribution.
Because these learned latents naturally align with actual physical dimensions, the universal policy can instantly adapt its control strategy to entirely new robot morphologies, achieving zero-shot transfer to unseen hardware.
Key Foundational Work
Hardware Conditioned Policies for Multi-Robot Transfer Learning (2018) — contributed 2 concept elements to this direction.
Unsupervised Learning of Latent Physical Properties Using Perception-Prediction Networks (2018) — contributed 2 concept elements to this direction.
Generalizable Features From Unsupervised Learning (2017) — contributed 2 concept elements to this direction.