Imagine you’re trying to figure out if a sealed opaque container is full of rice or pasta. You wouldn't just look at it; you’d pick it up, shake it, and listen to the sound. Robots can do this too, but there’s a catch: every robot "feels" and "hears" differently based on its specific motors, sensors, and arm structure (its morphology).
In a new paper, researchers from Tufts University and Bosch AI present a framework that allows a "newbie" robot to bootstrap its understanding of object properties by leveraging the "experience" of a veteran robot, even if their physical builds are completely different.
TL;DR
This paper introduces a framework to transfer implicit knowledge of non-visual object properties (like weight and content) between heterogeneous robots. By using manifold alignment and encoder-decoder networks, a newly deployed robot can learn to identify objects and their properties significantly faster than learning from scratch.
What Problem Does This Paper Address?
Interactive object perception—learning about objects by touching, grasping, and shaking them—is a slow, data-hungry process. Currently, if you deploy a new robot model, it usually has to start its "sensory education" from zero.
The challenge is that sensory data is embodiment-specific. A force sensor on a Baxter robot’s gripper produces different data distributions than a similar sensor on a UR5 arm. Because the "language" of their sensors is different, you can't simply copy-paste a machine learning model from one to the other. This paper tackles the "cross-robot transfer" problem: how do we translate the "feel" of an object from Robot A to Robot B?
Methods
The authors propose a multi-stage projection framework to bridge the gap between a source robot (experienced) and a target robot (newly deployed). They compare two primary mathematical approaches:
- Encoder-Decoder Network (EDN): A "translation" approach. An encoder turns the source robot's features into a latent code, and a decoder attempts to "reconstruct" what those features would look like if they had been recorded by the target robot.
- Kernel Manifold Alignment (KEMA): A "common language" approach. Instead of translating one to the other, KEMA projects features from both robots into a shared, latent mathematical space where similar object properties (e.g., "heavy" or "contains marbles") cluster together regardless of which robot did the shaking.
They also explore two types of correspondences:
- Object-identity: Mapping interactions with the exact same physical object.
- Object-property: Mapping interactions with different objects that share a property (e.g., two different red bowls that both weigh 100g).
Finally, they introduce a data augmentation technique that samples additional trials based on the mean and standard deviation of existing data to prevent overfitting and improve generalization.
Datasets
The study uses a substantial real-world dataset (which the authors have released):
- Robots: A 7-DOF Baxter and a 6-DOF UR5.
- Objects: 95 cylindrical containers varying in color (5), content (6 types like rice, marbles, or nuts & bolts), and weight (4 levels).
- Interactions: 7,600 total interactions across 8 behaviors (grasp, shake, push, drop, etc.).
- Sensors: Audio (microphones), Effort (joint torque), and Force (end-effector sensors).
Key Insights and Ideas
- Shared Spaces Win: KEMA (the shared latent space) consistently outperformed EDN (direct translation). This suggests that finding a "universal" representation of an object property is more effective than trying to simulate one robot's sensors using another's.
- Identity over Property: Building correspondences based on specific object identities (Robot A and B both touched "Object #42") creates a much better alignment than just matching general properties (Robot A touched "a heavy object" and Robot B touched "a different heavy object").
- Better than "Self-Knowledge": In some identity recognition tasks, the target robot actually performed better using projected features from the source robot than it did using its own limited raw data. This indicates that the transferred knowledge provides a cleaner, more generalized signal.
Key Results
- Property Recognition: In weight and content recognition, both transfer methods significantly outperformed the baseline (learning from scratch) when the target robot had limited experience (interacting with only a few objects).
- Identity Recognition: KEMA achieved a mean accuracy delta ($m\Delta A$) of -40.67 for the UR5 robot. In this context, a negative number is excellent—it means the error rate dropped significantly compared to the baseline.
- Efficiency: The framework allows a robot to achieve high recognition accuracy with only a fraction of the usual required physical interactions, effectively "skipping" the most time-consuming part of robot training.
Limitations and Future Work
While the results are promising, the authors note a few hurdles:
- Shared Context: The current model assumes both robots are performing the same behaviors in the same environment. Future work could explore transferring knowledge when robots use different exploratory behaviors.
- Autonomous Selection: Currently, the robots perform a fixed sequence of behaviors. The authors want to move toward "active learning," where the robot autonomously chooses which behavior (e.g., "should I shake this or drop it?") would most quickly resolve its uncertainty.
- Scaling Up: The study focused on two robots. The ultimate goal is a "fleet learning" scenario where dozens of different robot morphologies contribute to a massive, shared library of object property knowledge.
Summary updated