When a human picks up a mysterious box, they don't just rely on their eyes. They might give it a small shake to feel its weight, or a gentle nudge to see how easily it slides across the table. This is because vision is fundamentally limited: it can tell us about an object's geometry and color, but it cannot see mass, friction, stiffness, or the center of gravity.
In robotics, this "vision gap" is a primary hurdle for manipulation. A robot might see a water drum, but it doesn't know if it's empty or full until it tries to move it. To bridge this gap, a growing body of research focuses on interactive inference—the ability of a robot to estimate hidden physical properties in real-time by observing how objects respond to its own actions.
The Interaction-Observation-Update Loop
The core intuition behind this research is that physical properties are "latent" variables that reveal themselves through dynamics. If a robot applies a known force and the object moves less than expected, the robot can infer that the mass or friction is higher than initially thought.
Jiajun Wu, Joshua B. Tenenbaum, and their colleagues demonstrated this in DensePhysNet. Their system uses a predictive model to compare expected motion against observed outcomes. By iteratively refining its estimate of attributes like mass and friction through multi-step interactions (like sliding and colliding), the robot builds a "dense" map of the object's physics.
This approach transforms manipulation from a static planning problem into a dynamic estimation problem. As Alina Kloss and Jiajun Wu 0001 show in Accurate Vision-based Manipulation through Contact Reasoning, even simple recursive filtering can allow a robot to adapt to objects with non-uniform mass distributions by observing how they rotate when pushed.
Encoding Physics into Latent Context
How should a robot represent these "hidden" properties? While some systems attempt to estimate explicit values (like "0.5 kg"), many modern approaches use deep latent embeddings—mathematical vectors that capture the "essence" of an object's physics without needing human-readable labels.
In COCOI: Contact-aware Online Context Inference, Zhuo Xu and Masayoshi Tomizuka developed a parallel inference module. This module processes a history of past interactions to generate a latent representation of an object's physics. This "context" is then fed into a reinforcement learning policy, allowing the robot to push unstable 3D objects without knocking them over by "sensing" their stability on the fly.
Similarly, for complex tasks like dense packing, Bo Ai and Yunzhu Li introduced RoboPack. Their architecture combines Graph Neural Networks (GNNs) and LSTMs to process visual and tactile feedback. By "feeling" the resistance of objects in a cluttered space, the robot can infer hidden properties like softness or weight distribution, enabling it to squeeze objects into tight gaps where vision alone would fail.
Whole-Body Adaptation and Open Worlds
The need for interactive inference is most acute when robots move out of the lab and into the "open world," where they encounter objects with unpredictable mechanical behaviors like spring-loaded doors or heavy furniture.
Researchers at KAIST, including Seunghun Jeon and Jemin Hwangbo, applied this to quadrupedal robots in Learning Whole-body Manipulation for Quadrupedal Robot. By encoding interaction history into a latent space, a four-legged robot can "feel" the weight of a 19kg water drum and adjust its stance and force in real-time to move it, despite having no prior knowledge of the drum's mass.
This theme of "open-world" adaptation is echoed by Deepak Pathak and his team in their work on Adaptive Mobile Manipulation. They argue that because visual data cannot capture properties like spring tension in a cabinet door, robots must rely on interactive feedback. By combining imitation learning with online reinforcement learning, their system learns to master diverse articulated objects by simply trying to move them and learning from the resulting tactile and visual "surprise."
From Rigid Blocks to Messy Materials
The frontier of this research is moving toward "messy" materials—deformable objects, fluids, or granular piles. Yunzhu Li and Kris Hauser introduced AdaptiGraph, which uses graph-based neural dynamics to estimate hidden traits like stiffness or mass distribution in real-time. By performing brief, purposeful "probes," the robot can optimize its internal model to match what it sees, allowing for high-precision control of materials that don't follow simple rigid-body physics.
Ultimately, these papers suggest a shift in robotic intelligence: moving away from robots that "know" the world through pre-programmed models, and toward robots that "learn" the world through active, physical inquiry. As Joni Pajarinen and Ville Kyrki framed it in their POMDP approach, every action a robot takes is a trade-off between reaching a goal and gathering information. By treating physical properties as something to be discovered through action, robots become significantly more capable of handling the uncertainty of the real world.