Active Exploration for Robotic Manipulation

Tim Schneider, Boris Belousov, Georgia Chalvatzaki, Diego Romeres, Devesh K. Jha, Jan Peters

TL;DR

This paper introduces a model-based reinforcement learning framework that uses "active exploration" to solve complex robotic manipulation tasks where rewards are extremely rare. By incorporating an information-theoretic objective (Mutual Information or Lautum Information) into a Model Predictive Control (MPC) loop, the robot learns to be "curious" about its environment, allowing it to master challenging tasks like balancing and pushing a ball on a tilted table from scratch, both in simulation and on a real robot.

What Problem Does This Paper Address?

Robotic manipulation is hard because the physics of contact are complex and often unknown. Most current Reinforcement Learning (RL) methods struggle with "sparse rewards"—situations where the robot only gets a "win" signal when it perfectly completes a task (e.g., getting a ball into a tiny hole).

If a robot doesn't stumble upon the reward by accident, it never learns. Standard exploration (like adding random noise to actions) is often too aimless to solve these problems. The authors address this by making the robot's exploration directed: the agent actively seeks out states where it is most uncertain about the environment's dynamics, effectively "learning how to learn" before it even knows where the goal is.

Methods

The authors propose an information-seeking Model Predictive Control (MPC) approach. Key components include:

  • Probabilistic Ensembles: The robot learns a world model using an ensemble of neural networks. The disagreement or uncertainty between these networks represents "epistemic uncertainty"—what the robot knows it doesn't know.
  • Information Gain Objective: The planning objective is modified to maximize: Expected Reward + β * Information Gain. They explore two metrics for information gain: Mutual Information (MI) and the less common Lautum Information (LI).
  • Nested Monte Carlo (NMC) Estimator: Since information gain is notoriously hard to compute in real-time, they use a sample-reusing NMC estimator to make it computationally feasible for an MPC loop.
  • CEM with Memory Buffer: They identify a failure mode called "detachment," where a robot stops exploring because the area around its starting point is already well-understood. To fix this, they modify the Cross-Entropy Method (CEM) planner with a memory buffer of past successful trajectories to "warm-start" the search for new frontiers.

Datasets

The authors designed three custom benchmarks specifically for hard-to-explore continuous control:

  1. Tilted Pushing (Simulation): Pushing a ball up a slanted surface to a target zone.
  2. Tilted Pushing Maze (Simulation): A more complex version with holes that trap the ball, requiring the robot to navigate around obstacles.
  3. Tilted Pushing Real: The same task implemented on a physical 7-DOF robot arm using motion capture for tracking.

These tasks are challenging because the robot must constantly balance the ball while moving it, and the reward is only given at the very end.

Key Insights and Ideas

  • Lautum Information (LI) vs. Mutual Information (MI): One of the most interesting findings is that LI is significantly more sample-efficient than MI for these tasks. LI encourages the agent to seek observations that disprove its current model, which seems to drive more effective exploration in manipulation.
  • Solving "Detachment": The authors show that curiosity-driven agents often get stuck. Once the robot understands its immediate surroundings, the "curiosity reward" disappears. Without a global goal, it just sits still. Their "warm-start" memory buffer allows the robot to remember how to reach the edge of its knowledge and keep pushing forward.
  • Real-World Learning from Scratch: Unlike many papers that use "Sim-to-Real" (training in a simulator and transferring to a robot), this method is efficient enough to learn directly on the physical hardware in about 21 hours of training.

Key Results

  • Baseline Failure: Standard state-of-the-art algorithms like PETS, SAC, and MBPO failed to solve the tilted pushing tasks entirely because they couldn't find the sparse reward.
  • Success Rate: The proposed active exploration method solved the Tilted Pushing task consistently within 300,000 steps and the Maze task within 1.5 million steps.
  • Real-World Performance: The real robot successfully learned to balance and push the ball to the target, demonstrating that the information-theoretic objective translates well to the noisy, non-Markovian realities of physical contact (like ball spin).

Limitations and Future Work

  • Computational Weight: The MPC planner is computationally heavy, limiting the control frequency to 4 Hz. This makes it unsuitable for very high-speed, dynamic tasks (like catching a ball).
  • Hyperparameter Sensitivity: The weight of the curiosity term ($\beta$) needs careful tuning. If it's too high, the robot ignores the goal; if it's too low, it never explores. The authors suggest an adaptive $\beta$ but acknowledge this is an open research area.
  • Sensing: The current system relies on motion capture. The authors plan to incorporate tactile sensors (touch) in the future to provide richer feedback about object properties like friction and spin.

Summary updated

Abstract

Robotic manipulation stands as a largely unsolved problem despite significant advances in robotics and machine learning in recent years. One of the key challenges in manipulation is the exploration of the dynamics of the environment when there is continuous contact between the objects being manipulated. This paper proposes a model-based active exploration approach that enables efficient learning in sparse-reward robotic manipulation tasks. The proposed method estimates an information gain objective using an ensemble of probabilistic models and deploys model predictive control (MPC) to plan actions online that maximize the expected reward while also performing directed exploration. We evaluate our proposed algorithm in simulation and on a real robot, trained from scratch with our method, on a challenging ball pushing task on tilted tables, where the target ball position is not known to the agent a-priori. Our real-world robot experiment serves as a fundamental application of active exploration in model-based reinforcement learning of complex robotic manipulation tasks.
BibTeX
@inproceedings{schneider2022active,
  author = {Tim Schneider and Boris Belousov and Georgia Chalvatzaki and Diego Romeres and Devesh K. Jha and Jan Peters},
  title = {{Active Exploration for Robotic Manipulation}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2022},
  doi = {10.1109/iros47612.2022.9982061},
}

Loading related content...

Figure