Interactive Learning of Physical Object Properties Through Robot Manipulation and Database of Object Measurements

Andrej Kruzliak, Jiri Hartvich, Shubhan P. Patni, Lukas Rustler, Jan Kristof Behrens, Fares J. Abu-Dakka, Krystian Mikolajczyk, Ville Kyrki, Matej Hoffmann

Abstract

While computer vision has advanced significantly in object identification, estimating intrinsic physical properties like mass and stiffness remains a challenge for non-interactive systems. This paper introduces a framework that enables robots to autonomously "digitize" these properties through active manipulation. By utilizing a Bayesian Network to model property dependencies and selecting actions based on Expected Information Gain, the robot efficiently learns about its environment. We demonstrate the system's effectiveness on a set of household objects, including "adversarial" items designed to deceive vision-only systems, and introduce the IPALM database—a large-scale resource of robotic object measurements.

Introduction

Modern AI excels at identifying objects from visual data, yet it often fails to grasp hidden physical characteristics. For instance, a glass wineglass and a plastic one may appear identical to a camera but require different handling strategies. To truly understand an object, a robot must interact with it. However, physical interaction is costly in terms of time and mechanical wear. This research addresses the challenge of how a robot can decide which sequence of actions—such as poking, lifting, or squeezing—will most efficiently reveal an object's hidden physical properties.

Methodology

Bayesian Network Model

The core of the proposed framework is a Bayesian Network (BN) that represents the probabilistic dependencies between five key object properties: Category, Material, Elasticity, Density, and Volume.

The Bayesian Network model representing dependencies between object properties and their relation to robotic actions.

As shown in the model, knowing the "Category" (e.g., a bottle) provides a prior for "Volume," while "Material" (e.g., ceramic) informs "Density" and "Elasticity." The robot can perform five distinct actions to gather evidence:

  • Vision: Estimating category or material from images.
  • Sound: Analyzing the acoustic response from poking the object.
  • Squeezing: Measuring the stress-strain curve to estimate elasticity.
  • Weighing: Determining mass by lifting the object.

Active Action Selection

Rather than following a fixed sequence, the robot employs an active learning strategy called ACTSEL. It calculates the Expected Information Gain (IG) for every available action by simulating potential outcomes. The robot then selects the action predicted to most significantly reduce the uncertainty (entropy) of the target property. After each action, the robot updates its "belief" across the entire network. For example, a weight measurement refines the mass estimate and, through the BN, propagates updates to material and density beliefs.

Experimental Setup and Datasets

The framework was evaluated using a robotic setup and a diverse set of objects.

The experimental setup featuring a robotic arm and a variety of household objects, including adversarial items like real and plastic bananas.

The study utilized two primary data resources:

  1. Experimental Set: 17 household objects, including items from the YCB dataset and "adversarial" objects such as real vs. plastic bananas and soft vs. hard sponges.
  2. The IPALM Database: A massive open-access database introduced by the authors, containing over 24,000 measurements of 63 different objects collected using four different robotic grippers.

Key Results and Insights

  • Cross-Modal Synergy: The robot demonstrated intelligent behavior, such as choosing to Weigh an object to learn about its Material. Since density is a strong material indicator and mass depends on density and volume, the robot "realized" that lifting provided more reliable information than visual or acoustic cues alone.
  • Efficiency and Accuracy: The ACTSEL method significantly outperformed random action selection, often achieving a dramatic drop in uncertainty within the first two actions. The system successfully identified "trick" objects; for example, if a vision-based "banana" label suggested softness but squeezing revealed high stiffness, the BN correctly shifted the material belief to "hard plastic."
  • Intelligent Termination: The system avoids redundant actions by terminating the experiment when no remaining action is expected to provide a positive information gain.

Limitations and Future Work

Current limitations include sensor thresholds, such as a 95 kPa cap on stiffness measurements due to gripper resolution. Additionally, while Expectation-Maximization (EM) was used for efficient real-time inference, it occasionally faced numerical stability issues. Future work will explore adaptive exploration strategies using reinforcement learning to further refine how robots extract physical properties from their environment.

Summary updated

BibTeX
@inproceedings{kruzliak2024interactive,
  author = {Andrej Kruzliak and Jiri Hartvich and Shubhan P. Patni and Lukas Rustler and Jan Kristof Behrens and Fares J. Abu-Dakka and Krystian Mikolajczyk and Ville Kyrki and Matej Hoffmann},
  title = {{Interactive Learning of Physical Object Properties Through Robot Manipulation and Database of Object Measurements}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2024},
  doi = {10.1109/iros58592.2024.10802249},
}
Figure