Abstract
This paper introduces LatentIM, a robotic exploration framework that autonomously develops a staged curriculum of sensorimotor skills. By clustering a latent space that fuses object perceptions, actions, and outcomes, the system identifies distinct interaction regions. The robot then prioritizes exploration in regions where it demonstrates the highest learning progress, effectively mimicking developmental stages observed in human infants. Experimental results in a simulated environment show that LatentIM significantly outperforms traditional intrinsic motivation baselines in predictive accuracy.
Introduction
Equipping robots with sensorimotor skills—the ability to understand how physical actions affect objects—is a fundamental challenge. Traditionally, this requires massive amounts of data, which is expensive to collect. The core bottleneck is efficient exploration: random exploration wastes time on impossible tasks or redundant information. While intrinsic motivation (IM) has been used to guide robots toward novelty, most existing methods focus on isolated aspects of the environment, such as novel objects or actions. This paper argues that for a robot to truly understand its world, it must learn the relationship between the Object it sees, the Action it takes, and the resulting Outcome.
Methods
The proposed LatentIM system operates in three distinct phases to guide autonomous learning:
- Bootstrapping: The robot performs a short burst of random interactions to collect initial data, recording the object (top-down depth image), the action (parameterized arm movements), and the outcome (change in object state).
- Latent Space Formation: A Variational Autoencoder (VAE) compresses these high-dimensional "triplets" (Object-Action-Outcome) into a low-dimensional latent space. This space is then partitioned into distinct regions using a Gaussian Mixture Model (GMM).
- Active Exploration: Each region is assigned its own Forward Model (a neural network predicting outcomes). The robot uses an IM signal called Learning Progress (LP)—the rate at which a model’s prediction error drops—to choose which region to explore next.
Key Insights and Developmental Staging
A critical insight of this work is that clustering in a blended latent space (Object + Action + Outcome) creates semantically meaningful categories. The robot does not just group objects; it groups interaction types, such as "lifting a cup with a pinch grasp" versus "pushing a cylinder."
By following Learning Progress, the robot naturally avoids "boring" regions (already mastered) and "impossible" regions (too noisy to learn), staying in a "Goldilocks" zone of maximum information gain. This leads to a staged curriculum where the robot prioritizes basic grasp outcomes before moving to complex purposeful actions, mirroring human infant development.
Results
LatentIM was evaluated in a CoppeliaSim environment using a UR10 robot arm and a variety of objects (cups, cylinders, spheres). The system significantly outperformed baseline models that used only one component (e.g., Object-only or Action-only IM) or random exploration.
As shown in the results, LatentIM achieved a much lower Mean Squared Error (MSE) in predicting action outcomes. After just 50 exploration steps, the performance gap between LatentIM and other methods became statistically significant ($p < 0.001$). Furthermore, the five regions formed in the latent space corresponded to distinct physical primitives, such as non-lifting pinch-grasps and lifting power-grasps.
Conclusion and Future Work
LatentIM demonstrates that fusing objects, actions, and outcomes into a shared latent space allows for more efficient and semantically grounded autonomous exploration. While the current study used static regions, future work will explore dynamic region creation (assimilation and accommodation) to support lifelong learning in more complex, cluttered environments.
Summary updated