Uncertainty-aware Contact-safe Model-based Reinforcement Learning

Cheng-Yu Kuo, Andreas Schaarschmidt, Yunduan Cui, Tamim Asfour, Takamitsu Matsubara

Introduction

Model-based Reinforcement Learning (MBRL) has emerged as a powerful paradigm for robotics due to its sample efficiency. By learning a "world model" of environment dynamics, robots can simulate experiences and plan actions without relying solely on exhaustive trial and error. However, a significant bottleneck remains in "contact-rich" environments—such as a kitchen where a robot must stir a bowl or scoop ingredients.

During the early stages of learning, the world model is often inaccurate. In these scenarios, standard MBRL can produce aggressive, unpredictable movements that lead to high-intensity collisions. While traditional RL treats these failures as data points, in the real world, smashing a spoon into a bowl at high speed can break hardware. This research introduces a "contact-safe" approach that links a robot's control limits directly to its model uncertainty, effectively forcing the robot to "slow down" when it is unsure of its surroundings.

Comparison of contact-unsafe behaviors versus the proposed contact-safe MBRL framework which scales control limits based on uncertainty.

Methodology: Uncertainty as a Safety Governor

The core innovation of this work is the association of model uncertainty with dynamic control limits. The framework operates through four key components:

  1. Dynamics Modeling (LGM-FF): To maintain real-time control frequencies, the authors utilize Linear Gaussian Model with Fastfood random features (LGM-FF). This approximation of Gaussian Processes allows the robot to predict both the next state and the associated variance (uncertainty) efficiently.
  2. Deterministic pMPC: The system uses a probabilistic Model Predictive Control (pMPC) approach, reformulated into a deterministic version to handle dynamic constraints without the computational overhead of standard probabilistic solvers.
  3. Uncertainty-Aware Control Limits: The controller modifies the allowed acceleration ($u$) based on the current model uncertainty. This is achieved through two mechanisms: Scaling, which shrinks the range of allowed actions as uncertainty grows, and Translating, which shifts the control limits to bias the robot toward "braking" or reducing velocity when confused.
  4. Delay Compensation: A "one-step ahead" planning strategy ensures that control signals remain synchronized with the robot's actual state, accounting for the time required to solve the MPC.
The scaling and translating mechanism adjusts the original control limits ($u_{min}, u_{max}$) into contact-safe limits based on the predicted mean and variance.

Experimental Evaluation

The approach was evaluated across three distinct robotic manipulation tasks designed to test both simulation accuracy and real-world robustness:

  • Simulated Mixing: A 2-DOF arm in a MuJoCo environment stirring 900 particles.
  • Real Mixing: A physical HEBI robotic arm performing the same task with a metal bowl and cut straws.
  • Real Scooping: A 4-DOF HEBI arm learning to transfer straws between bowls, testing the method's scalability to higher-dimensional state spaces.

Key Results and Insights

The "contact-safe" MBRL demonstrated that safety does not have to come at the expense of learning speed. Key findings include:

  • Force Reduction: In simulations, the framework reduced maximum contact forces by 50-70% during early learning trials compared to standard MBRL.
  • Efficiency Preservation: Despite the cautious behavior, the robot acquired tasks in roughly the same number of trials as unsafe baselines. In the mixing task, both methods achieved stable tracking error (under 2cm) by the 8th trial.
  • Operational Cleanliness: In physical tests, the contact-safe robot scattered significantly fewer ingredients outside the bowl, proving that its tentative movements in high-uncertainty zones led to more controlled task execution.

Conclusion and Future Work

By treating uncertainty as a safety governor rather than just a driver for exploration, this work mimics the human tendency to move slowly in unfamiliar environments. While the current model assumes independent state dimensions and zero-centered limits, future work aims to capture correlations between dimensions and adapt the framework for tasks requiring constant force, such as gravity compensation for heavy objects. This research marks a significant step toward deploying autonomous learning systems in fragile, real-world environments.

Summary updated

BibTeX
@article{kuo2021uncertainty,
  author = {Cheng-Yu Kuo and Andreas Schaarschmidt and Yunduan Cui and Tamim Asfour and Takamitsu Matsubara},
  title = {{Uncertainty-aware Contact-safe Model-based Reinforcement Learning}},
  journal = {IEEE Robotics Autom. Lett.},
  year = {2021},
  url = {https://openreview.net/forum?id=zE1jxMGEyev},
}
Figure