Simulation-Aided Policy Tuning for Black-Box Robot Learning

Shiming He, Alexander von Rohr, Dominik Baumann, Ji Xiang, Sebastian Trimpe

Abstract

How can robots learn and adapt to new tasks and situations with little data? Systematic exploration and simulation are crucial tools for efficient robot learning. We present a novel black-box policy search algorithm focused on data-efficient policy improvements. The algorithm learns directly on the robot and treats simulation as an additional information source to speed up the learning process. At the core of the algorithm, a probabilistic model learns the dependence of the policy parameters and the robot learning objective not only by performing experiments on the robot, but also by leveraging data from a simulator. This substantially reduces interaction time with the robot. Using this model, we can guarantee improvements with high probability for each policy update, thereby facilitating fast, goal-oriented learning. We evaluate our algorithm on simulated fine-tuning tasks and demonstrate the data-efficiency of the proposed dual-information source optimization algorithm. In a real robot learning experiment, we show fast and successful task learning on a robot manipulator with the aid of an imperfect simulator.

Introduction

Training robots in the real world is notoriously difficult. Standard Reinforcement Learning (RL) often requires thousands of trials, leading to significant mechanical wear and tear. While simulators offer a fast and safe alternative, they suffer from the "reality gap"—discrepancies between simulated physics and real-world dynamics mean a policy perfected in simulation often fails on a real robot.

Conceptual overview of simulation-aided policy improvement.

This paper introduces S-HCI-GIBO, a data-efficient black-box optimization algorithm that speeds up robot learning by intelligently combining "cheap" but imperfect simulations with "expensive" real-world experiments. The method treats simulation not as a replacement for reality, but as a biased yet useful source of information to guide local policy tuning, specifically addressing the challenge of high-dimensional, data-efficient robot learning.

Methodology

The authors build upon GIBO (Gradient Information with BO), a local search method that uses Gaussian Processes (GPs) to infer the gradient of a performance function from noisy observations. They introduce two major innovations:

  1. HCI-GIBO (High-Confidence Improvement GIBO): Instead of taking a fixed number of samples to estimate a gradient, this version uses a mathematical criterion to query the objective only until it is "confident enough" (e.g., 95% probability) that the next update step will actually improve the policy. This prevents wasting real-world trials on redundant data.
  2. S-HCI-GIBO (Simulation-Aided HCI-GIBO): This extends the model to a multi-fidelity setting. It uses a "Sim-to-Real switching rule" that exhausts useful information from the simulator first. If the simulator can no longer provide enough information to guarantee a high-probability improvement on the real robot, the algorithm then switches to querying the physical hardware.
Step-by-step visualization of the S-HCI-GIBO algorithm.

Key Insights

  • Gradients over Values: The core insight is that the simulator doesn't need to accurately predict the total reward. It only needs to provide a useful direction (gradient). Even a biased simulator can often indicate how to move parameters to improve results.
  • Active Sampling for Confidence: By replacing a fixed sampling budget with a probability-of-improvement threshold, the algorithm becomes "frugal," stopping sampling as soon as the next step is a safe bet.
  • Hierarchical Information Sources: Sim data is treated as "free" and Robot data as "expensive." By modeling the correlation between the two, the robot only moves when the simulator has "run out of ideas."

Experiments and Results

The researchers evaluated their approach across three distinct tiers:

  • Synthetic Benchmarks: High-dimensional functions (up to 52D) sampled from known GP priors to test scalability.
  • Simulated Robot Tasks: Complex environments including Hopper Jump (15D), 5D Reacher (25D), and Box Pushing (70D).
  • Real-World Hardware: A Franka Emika robot arm tasked with balancing a planar pendulum while following a "Lissajous curve" trajectory, tuning 24 weights of a Dynamic Movement Primitive (DMP).

Key Results

  • High-Dimensional Success: In the 70-dimensional Box Pushing task, S-HCI-GIBO significantly outperformed standard RL baselines and other BO variants in convergence speed and final reward.
  • Real-World Efficiency: In the pendulum balancing experiment, S-HCI-GIBO reached 80% of the maximum reward using less than half the real-world trials required by the version without simulation aid (HCI-GIBO).
  • Robustness to Bias: The method successfully learned on the real Franka robot even with an "imperfect" simulator, demonstrating that the dual-GP model effectively manages the reality gap.

Conclusion and Future Work

S-HCI-GIBO provides a robust framework for data-efficient robot learning by leveraging imperfect simulations. While powerful, the method is a local search and can get stuck in local optima, necessitating a decent initialization. Future work aims to extend this to multiple information sources and explicitly handle safety constraints, such as ensuring the robot avoids collisions during exploration.

Summary updated

Abstract

How can robots learn and adapt to new tasks and situations with little data? Systematic exploration and simulation are crucial tools for efficient robot learning. We present a novel black-box policy search algorithm focused on data-efficient policy improvements. The algorithm learns directly on the robot and treats simulation as an additional information source to speed up the learning process. At the core of the algorithm, a probabilistic model learns the dependence of the policy parameters and the robot learning objective not only by performing experiments on the robot, but also by leveraging data from a simulator. This substantially reduces interaction time with the robot. Using this model, we can guarantee improvements with high probability for each policy update, thereby facilitating fast, goal-oriented learning. We evaluate our algorithm on simulated fine-tuning tasks and demonstrate the data-efficiency of the proposed dual-information source optimization algorithm. In a real robot learning experiment, we show fast and successful task learning on a robot manipulator with the aid of an imperfect simulator.
BibTeX
@article{he2025simulation,
  author = {Shiming He and Alexander von Rohr and Dominik Baumann and Ji Xiang and Sebastian Trimpe},
  title = {{Simulation-Aided Policy Tuning for Black-Box Robot Learning}},
  journal = {IEEE Trans. Robotics},
  year = {2025},
  doi = {10.1109/tro.2025.3539192},
}
Figure