Active Exploration for Robotic Manipulation

Tim Schneider, Boris Belousov, Georgia Chalvatzaki, Diego Romeres, Devesh K. Jha, Jan Peters

Abstract

Robotic manipulation stands as a largely unsolved problem despite significant advances in robotics and machine learning in recent years. One of the key challenges in manipulation is the exploration of the dynamics of the environment when there is continuous contact between the objects being manipulated. This paper proposes a model-based active exploration approach that enables efficient learning in sparse-reward robotic manipulation tasks. The proposed method estimates an information gain objective using an ensemble of probabilistic models and deploys model predictive control (MPC) to plan actions online that maximize the expected reward while also performing directed exploration. We evaluate our proposed algorithm in simulation and on a real robot, trained from scratch with our method, on a challenging ball pushing task on tilted tables, where the target ball position is not known to the agent a-priori. Our real-world robot experiment serves as a fundamental application of active exploration in model-based reinforcement learning of complex robotic manipulation tasks.

Introduction

Robotic manipulation is inherently difficult due to the complex and often unknown physics of contact. Most current Reinforcement Learning (RL) methods struggle with "sparse rewards"—situations where the robot only receives a success signal upon perfect completion of a task, such as guiding a ball into a small target zone.

Simulation and real-world experimental setups for the tilted pushing task.

Without encountering the reward by chance, standard RL agents fail to learn. Aimless exploration, such as adding random noise to actions, is typically insufficient for these high-dimensional contact tasks. This research addresses the problem by making exploration directed: the agent actively seeks out states where its internal model of the environment's dynamics is most uncertain, effectively "learning how to learn" before the goal is even located.

Methodology: Information-Seeking MPC

The core of the proposed framework is an information-seeking Model Predictive Control (MPC) approach. Unlike standard MPC which only optimizes for a task reward, this method incorporates a "curiosity" term.

Probabilistic Ensembles

The robot learns a world model using an ensemble of neural networks. The disagreement or variance between these networks represents "epistemic uncertainty"—the gap in the robot's knowledge about the environment.

Information Gain Objective

The planning objective is modified to maximize: Expected Reward + β * Information Gain. The authors evaluate two metrics for information gain: Mutual Information (MI) and Lautum Information (LI). To make this computationally feasible for real-time MPC, they utilize a sample-reusing Nested Monte Carlo (NMC) estimator.

Solving "Detachment"

A common failure in curiosity-driven agents is "detachment," where a robot stops exploring because the immediate vicinity is well-understood. To overcome this, the authors modify the Cross-Entropy Method (CEM) planner with a memory buffer of past successful trajectories. This "warm-starts" the search, allowing the robot to return to the frontiers of its knowledge.

Experimental Benchmarks

The method was evaluated on three custom benchmarks designed for hard-to-explore continuous control:

The three evaluation environments: tilted pushing (sim), tilted pushing maze (sim), and tilted pushing (real).
  1. Tilted Pushing (Simulation): Pushing a ball up a slanted surface to a target zone.
  2. Tilted Pushing Maze (Simulation): A complex version with holes that trap the ball, requiring navigation around obstacles.
  3. Tilted Pushing (Real): The task implemented on a physical 7-DOF robot arm using motion capture for tracking.

These tasks are challenging because the robot must balance the ball against gravity while moving it, and the reward is only provided at the final target.

Key Insights and Results

The Power of Lautum Information

A significant finding is that Lautum Information (LI) is more sample-efficient than Mutual Information (MI). LI encourages the agent to seek observations that disprove its current model, driving more effective exploration in manipulation tasks.

Real-World Learning from Scratch

While many robotic studies rely on "Sim-to-Real" transfer, this method is efficient enough to learn directly on physical hardware. The real robot mastered the task in approximately 21 hours of training from scratch.

Performance Comparison

Standard state-of-the-art algorithms (e.g., PETS, SAC, MBPO) failed to solve the tilted pushing tasks because they could not find the sparse reward. In contrast, the proposed active exploration method consistently solved the Tilted Pushing task within 300,000 steps and the Maze task within 1.5 million steps.

Limitations and Future Work

The primary limitation is the computational weight of the MPC planner, which limits control frequency to 4 Hz, making it unsuitable for high-speed dynamic tasks. Additionally, the curiosity weight (β) requires careful tuning. Future work aims to incorporate tactile sensing to provide richer feedback on object properties like friction and spin, reducing reliance on external motion capture systems.

Summary updated

Abstract

Robotic manipulation stands as a largely unsolved problem despite significant advances in robotics and machine learning in recent years. One of the key challenges in manipulation is the exploration of the dynamics of the environment when there is continuous contact between the objects being manipulated. This paper proposes a model-based active exploration approach that enables efficient learning in sparse-reward robotic manipulation tasks. The proposed method estimates an information gain objective using an ensemble of probabilistic models and deploys model predictive control (MPC) to plan actions online that maximize the expected reward while also performing directed exploration. We evaluate our proposed algorithm in simulation and on a real robot, trained from scratch with our method, on a challenging ball pushing task on tilted tables, where the target ball position is not known to the agent a-priori. Our real-world robot experiment serves as a fundamental application of active exploration in model-based reinforcement learning of complex robotic manipulation tasks.
BibTeX
@inproceedings{schneider2022active,
  author = {Tim Schneider and Boris Belousov and Georgia Chalvatzaki and Diego Romeres and Devesh K. Jha and Jan Peters},
  title = {{Active Exploration for Robotic Manipulation}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2022},
  doi = {10.1109/iros47612.2022.9982061},
}
Figure