Uncertainty-Aware Contact-Safe Model-Based Reinforcement Learning

Cheng-Yu Kuo, Andreas Schaarschmidt, Yunduan Cui, Tamim Asfour, Takamitsu Matsubara

TL;DR

This paper introduces a "contact-safe" approach to Model-based Reinforcement Learning (MBRL) that prevents robots from damaging themselves or their surroundings during the learning process. By linking the robot's control limits directly to its model uncertainty, the system automatically adopts cautious, low-velocity behaviors when it is unsure of the environment's dynamics, effectively "slowing down" until it learns enough to move confidently.

What Problem Does This Paper Address?

Model-based Reinforcement Learning (MBRL) is highly valued in robotics because it is sample-efficient—it learns a "world model" to simulate experiences rather than relying solely on trial and error. However, there is a catch: during the early stages of learning, the model is often wildly inaccurate due to a lack of data.

In "contact-rich" environments (like a kitchen where a robot must stir a bowl or scoop ingredients), these inaccurate models can lead to aggressive, unpredictable movements. A robot trying to learn how to stir might accidentally smash the spoon into the side of the bowl at high speed. While traditional RL uses these "failures" to learn, in the real world, such impacts can break hardware. The challenge is: How can a robot explore and learn contact-rich tasks without causing high-intensity, damaging collisions before it knows what it's doing?

Methods

The authors propose a framework that associates model uncertainty with control limits.

  1. Dynamics Modeling (LGM-FF): To keep the system computationally efficient for real-time control, they use an approximation of Gaussian Processes called Linear Gaussian Model with Fastfood random features (LGM-FF). This allows the robot to predict not just the next state, but also the variance (uncertainty) of that prediction.
  2. Deterministic pMPC: They use a probabilistic Model Predictive Control (pMPC) approach, reformulated into a deterministic version to handle dynamic constraints efficiently.
  3. Uncertainty-Aware Control Limits: This is the core innovation. They define two mechanisms that modify the robot's allowed acceleration ($u$) based on the current model uncertainty ($\Sigma_p$):
  4. Scaling ($K_s$): As uncertainty increases, the range of allowed control signals shrinks. This prevents the robot from taking "big" or aggressive actions when it doesn't trust its model.
  5. Translating ($K_t$): This shifts the control limits to favor actions that reduce the robot's velocity. Essentially, if the robot is confused, the controller is biased toward "braking."
  6. Delay Compensation: They implement a "one-step ahead" planning strategy to account for the computational time required to solve the MPC, ensuring the control signals remain consistent with the robot's actual state.

Datasets

The approach was evaluated on custom robotic manipulation tasks rather than standard static datasets:

  • Simulated Mixing Task: A 2-Degree-of-Freedom (DoF) arm in a MuJoCo environment stirring a bowl filled with 900 particles.
  • Real Mixing Task: A physical 2-DoF HEBI robotic arm performing the same stirring task with a metal bowl and cut straws.
  • Real Scooping Task: A 4-DoF HEBI arm learning to scoop straws from one bowl to another, demonstrating the method's scalability to higher dimensions and larger workspaces.

Key Insights and Ideas

  • Uncertainty as a Safety Governor: Instead of just using uncertainty to drive exploration (as many RL papers do), this work uses it as a safety governor. It mimics human behavior: we move slowly and tentatively when reaching into a dark or unfamiliar space.
  • Scaling vs. Translating: The authors found that simply limiting the range of motion (scaling) isn't enough. If a robot is already moving fast and suddenly hits a high-uncertainty zone, it needs a mechanism to actively slow down (translating) to avoid a high-impact collision.
  • Efficiency Matters: By using LGM-FF and a deterministic MPC reformulation, the authors achieved a control frequency high enough for real-world robotic tasks, which is often a bottleneck for GP-based MBRL.

Key Results

  • Reduced Impact Force: In simulations, the "contact-safe" MBRL reduced the maximum measured contact forces by roughly 50-70% compared to standard MBRL during the early trials.
  • Maintained Learning Efficiency: Despite being more cautious, the robot still acquired the tasks (mixing and scooping) in roughly the same number of trials as the "unsafe" baseline. In the mixing task, both reached a stable tracking error (under 2cm) by the 8th trial.
  • Real-World Robustness: In physical experiments, the contact-safe robot scattered significantly fewer "ingredients" (straws) outside the bowl, proving that its cautious movements led to cleaner, safer task execution during the learning phase.

Limitations and Future Work

  • Zero-Centered Limits: The current control limits are designed around an origin of zero. This works for velocity-controlled robots but might struggle with tasks requiring constant force to counteract gravity (like holding a heavy object) without additional compensation.
  • Independent Dimensions: The model currently treats each state dimension (e.g., X, Y, Z position) as independent. In complex tasks, these dimensions are often highly correlated; capturing these correlations could improve the model's accuracy.
  • Execution Safety: This paper focuses on safety during learning. The authors suggest that integrating specific safety measures for the execution phase (after the model is learned) would be a natural next step.

Summary updated

Abstract

This letter presents contact-safe Model-based Reinforcement Learning (MBRL) for robot applications that achieves contact-safe behaviors in the learning process. In typical MBRL, we cannot expect the data-driven model to generate accurate and reliable policies to the intended robotic tasks during the learning process due to sample scarcity. Operating these unreliable policies in a contact-rich environment could cause damage to the robot and its surroundings. To alleviate the risk of causing damage through unexpected intensive physical contacts, we present the contact-safe MBRL that associates the probabilistic Model Predictive Control's (pMPC) control limits with the model uncertainty so that the allowed acceleration of controlled behavior is adjusted according to learning progress. Control planning with such uncertainty-aware control limits is formulated as a deterministic MPC problem using a computation-efficient approximated GP dynamics and an approximated inference technique. Our approach's effectiveness is evaluated through bowl mixing tasks with simulated and real robots, scooping tasks with a real robot as examples of contact-rich manipulation skills.
BibTeX
@article{kuo2021uncertainty,
  author = {Cheng-Yu Kuo and Andreas Schaarschmidt and Yunduan Cui and Tamim Asfour and Takamitsu Matsubara},
  title = {{Uncertainty-Aware Contact-Safe Model-Based Reinforcement Learning}},
  journal = {IEEE Robotics Autom. Lett.},
  year = {2021},
  url = {https://openreview.net/forum?id=zE1jxMGEyev},
}
Figure