Online Inference of Hidden Physics through Robotic Interaction

This direction is merged into another direction: latent physical property estimation. Its last version stays readable and citable here.

Version 1 of 117 papers20 conceptsfrozen 2026-10-04

When a human picks up a mysterious box, they don't just rely on their eyes. They might give it a small shake to feel its weight, or a gentle nudge to see how easily it slides across the table. This is because vision is fundamentally limited: it can tell us about an object's geometry and color, but it cannot see mass, friction, stiffness, or the center of gravity.

In robotics, this "vision gap" is a primary hurdle for manipulation. A robot might see a water drum, but it doesn't know if it's empty or full until it tries to move it. To bridge this gap, a growing body of research focuses on interactive inference—the ability of a robot to estimate hidden physical properties in real-time by observing how objects respond to its own actions.

The Interaction-Observation-Update Loop

The core intuition behind this research is that physical properties are "latent" variables that reveal themselves through dynamics. If a robot applies a known force and the object moves less than expected, the robot can infer that the mass or friction is higher than initially thought.

Jiajun Wu, Joshua B. Tenenbaum, and their colleagues demonstrated this in DensePhysNet. Their system uses a predictive model to compare expected motion against observed outcomes. By iteratively refining its estimate of attributes like mass and friction through multi-step interactions (like sliding and colliding), the robot builds a "dense" map of the object's physics.

This approach transforms manipulation from a static planning problem into a dynamic estimation problem. As Alina Kloss and Jiajun Wu 0001 show in Accurate Vision-based Manipulation through Contact Reasoning, even simple recursive filtering can allow a robot to adapt to objects with non-uniform mass distributions by observing how they rotate when pushed.

Encoding Physics into Latent Context

How should a robot represent these "hidden" properties? While some systems attempt to estimate explicit values (like "0.5 kg"), many modern approaches use deep latent embeddings—mathematical vectors that capture the "essence" of an object's physics without needing human-readable labels.

In COCOI: Contact-aware Online Context Inference, Zhuo Xu and Masayoshi Tomizuka developed a parallel inference module. This module processes a history of past interactions to generate a latent representation of an object's physics. This "context" is then fed into a reinforcement learning policy, allowing the robot to push unstable 3D objects without knocking them over by "sensing" their stability on the fly.

Similarly, for complex tasks like dense packing, Bo Ai and Yunzhu Li introduced RoboPack. Their architecture combines Graph Neural Networks (GNNs) and LSTMs to process visual and tactile feedback. By "feeling" the resistance of objects in a cluttered space, the robot can infer hidden properties like softness or weight distribution, enabling it to squeeze objects into tight gaps where vision alone would fail.

Whole-Body Adaptation and Open Worlds

The need for interactive inference is most acute when robots move out of the lab and into the "open world," where they encounter objects with unpredictable mechanical behaviors like spring-loaded doors or heavy furniture.

Researchers at KAIST, including Seunghun Jeon and Jemin Hwangbo, applied this to quadrupedal robots in Learning Whole-body Manipulation for Quadrupedal Robot. By encoding interaction history into a latent space, a four-legged robot can "feel" the weight of a 19kg water drum and adjust its stance and force in real-time to move it, despite having no prior knowledge of the drum's mass.

This theme of "open-world" adaptation is echoed by Deepak Pathak and his team in their work on Adaptive Mobile Manipulation. They argue that because visual data cannot capture properties like spring tension in a cabinet door, robots must rely on interactive feedback. By combining imitation learning with online reinforcement learning, their system learns to master diverse articulated objects by simply trying to move them and learning from the resulting tactile and visual "surprise."

From Rigid Blocks to Messy Materials

The frontier of this research is moving toward "messy" materials—deformable objects, fluids, or granular piles. Yunzhu Li and Kris Hauser introduced AdaptiGraph, which uses graph-based neural dynamics to estimate hidden traits like stiffness or mass distribution in real-time. By performing brief, purposeful "probes," the robot can optimize its internal model to match what it sees, allowing for high-precision control of materials that don't follow simple rigid-body physics.

Ultimately, these papers suggest a shift in robotic intelligence: moving away from robots that "know" the world through pre-programmed models, and toward robots that "learn" the world through active, physical inquiry. As Joni Pajarinen and Ville Kyrki framed it in their POMDP approach, every action a robot takes is a trade-off between reaching a goal and gathering information. By treating physical properties as something to be discovered through action, robots become significantly more capable of handling the uncertainty of the real world.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_online_inference_of_hidden_physics_throu_39b31ece,
  title = {Online Inference of Hidden Physics through Robotic Interaction},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/online-inference-of-hidden-physics-through-robotic-interaction/rdr_39b31ecef6e4401bacf69aa04fed0f02},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_online_inference_of_hidden_physics_throu_39b31ece_v1,
  title = {Online Inference of Hidden Physics through Robotic Interaction},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/online-inference-of-hidden-physics-through-robotic-interaction/rdr_39b31ecef6e4401bacf69aa04fed0f02/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}