Imagine looking at two identical cardboard boxes sitting on a table. One is empty; the other is packed tightly with heavy lead weights. To a standard computer vision model, these boxes are completely indistinguishable. They share the exact same geometry, surface texture, color, and spatial coordinates. Yet, if a robotic arm attempts to push them with the same force, one will slide effortlessly across the table while the other will barely budge.
This discrepancy highlights a fundamental limitation in purely visual AI perception: standard vision systems excel at mapping explicit visual attributes but fail to capture the latent physical properties—such as mass, friction, elasticity, and internal mass distribution—that govern how objects actually behave (Jiafei Duan et al.; Iain Haughton et al.). Without a way to infer these hidden variables, models cannot accurately simulate future states, plan robust manipulation policies, or reason counterfactually about "what if" scenarios (Maitreya Patel et al.; Julien Mille).
Inverting Dynamics via Interaction
How do we see the invisible? The answer lies in dynamics. While a static image cannot reveal an object's mass, a video of that object interacting with its environment can.
Consider a billiard table. If a cue ball strikes a target ball and stops dead while the target ball flies forward, we intuitively know they have equal mass. If the cue ball bounces backward, the target ball is heavier. This is the core mechanism behind latent physical property estimation: using observed kinematics to invert the underlying equations of motion.
[Applied Forces] + [Latent Properties (Mass, Friction)] ---> Forward Physics Engine ---> [Observed Trajectory]
|
[Inferred Latent Properties] <--- Inverse Physics / Optimization <-----------------------------+
As Zhenfang Chen et al. demonstrate, intrinsic properties like mass or electric charge can be systematically disentangled by observing how objects interact over time. Instead of trying to guess these properties from surface textures, models can leverage neuro-symbolic architectures or learnable physics engines to bridge the gap between raw pixels and structured physical laws (Alvaro Sanchez-Gonzalez et al.; Akash Srivastava et al.).
However, this creates a circular dependency: to predict a trajectory, you need the physical laws and the parameters; to find the parameters, you need to understand the laws. The authors of BSP address this through a Bayesian-symbolic approach that simultaneously induces force laws and estimates hidden object attributes from visual data.
The Necessity of Temporal and Causal Context
A single frame or a brief snippet of video is often physically ambiguous. If a sliding block slows down rapidly, is it because it is heavy, or because the surface friction coefficient is high?
To resolve this ambiguity, models must look across entire temporal sequences to capture long-term causal dependencies. Xiangyu Zhu et al. show that hidden variables act as confounders in physical prediction tasks. By employing global attention mechanisms across time, models can deconfound these dynamics, allowing for highly accurate counterfactual predictions (e.g., "What would happen if this block were removed from the tower?").
Crucially, when objects are isolated, estimation is impossible. As Mingyu Ding points out, if an object never interacts or collides with anything else in a video sequence, its latent properties remain completely unidentifiable. The interaction is the catalyst that forces the hidden property to manifest in the observable state space.
Active Probing and the Power of Intervention
What if the environment does not naturally provide the interactions we need? If an object is just sitting still, a passive observer is helpless. This is where autonomous agents must shift from passive observation to active experimentation.
Humans do this naturally—we pick up an object and shake it to feel its weight distribution. In both digital and physical worlds, active intervention breaks the ambiguity of passive data. Neil Bramley et al. found that active learners isolate hidden variables (like magnetic forces or mass) far better than passive observers because they can design "natural experiments" that systematically vary one constraint at a time.
In robotics, this translates to controllers designed specifically for information gain. Saeid Amiri et al. utilize a Mixed Observability Markov Decision Process (MOMDP) to plan sequences of physical interactions—like poking, shaking, or lifting—to rapidly verify latent attributes based on a user's query. Similarly, Iain Haughton et al. demonstrate an autonomous robot experimenter that physically probes scenes to build dense, real-time 3D maps of rigidity and friction.
Beyond Vision: Multimodal Physical Inference
Vision is inherently a remote sensor; it only captures the light bouncing off a surface. To truly grasp the internal state of an object, an agent often needs to listen or feel.
Consider the difference between a full container and an empty one. When tapped, they sound completely different. Abhinav Gupta et al. introduced Tilt-Bot to exploit this exact phenomenon, showing that audio feedback provides a direct window into latent properties like internal liquid presence or mass distribution, outperforming vision-only models by a wide margin.
Touch provides another critical dimension. When packing objects tightly into clutter, visual occlusions make it impossible to see how an object is reacting. Bo Ai et al. show that by integrating a brief history of tactile feedback, a dynamics model can infer material stiffness and weight distributions on the fly, allowing robots to successfully execute force-sensitive manipulation tasks where vision alone fails.