Guided Uncertainty-Aware Policy Optimization: Combining Learning and Model-Based Strategies for Sample-Efficient Policy Learning

Michelle A. Lee, Carlos Florensa, Jonathan Tremblay, Nathan D. Ratliff, Animesh Garg, Fabio Ramos, Dieter Fox

TL;DR

This paper introduces GUAPO, a novel algorithm that combines the precision of model-based control with the adaptability of reinforcement learning (RL) for robotic tasks. By leveraging uncertainty estimates from perception systems, GUAPO intelligently switches between these two approaches, enabling sample-efficient learning and robust performance even with imperfect robot and sensor models.

What Problem Does This Paper Address?

Traditional robotics often relies on highly accurate models of the environment and precise perception systems. While model-based (MB) approaches are efficient in known environments, they are brittle and fail when perception or actuation is inaccurate. Reinforcement learning (RL) can handle raw sensory inputs and adapt to uncertainties, but it's notoriously sample-inefficient and can be unsafe during exploration. This paper addresses the challenge of creating robotic systems that are both sample-efficient and robust to real-world imperfections, bridging the gap between the strengths of MB and RL methods.

Methods

The core of GUAPO is a hybrid policy that dynamically switches between a model-based controller ($\pi_{MB}$) and a reinforcement learning policy ($\pi_{RL}$). The decision to switch is guided by uncertainty estimates from a perception system.

  • Uncertainty-Aware Perception: Instead of just providing a single object pose, the perception system (DOPE in their case) outputs an uncertainty estimate for the object's location. This defines a region of uncertainty, $\hat{S}_u$, where the MB controller might fail.
  • Hybrid Policy: The algorithm uses a switching function $\alpha(s)$ that is 0 when the robot is outside $\hat{S}_u$ (where the MB controller is trusted) and 1 when inside $\hat{S}_u$ (where the RL policy is activated). The overall policy is $\pi(a|s) = (1 - \alpha(s)) \cdot \pi_{MB}(a|s) + \alpha(s) \cdot \pi_{RL}(a|s)$.
  • Model-Based Controller: For regions outside $\hat{S}_u$, a model-based controller (Riemannian Motion Policies - RMPs) is used to guide the robot towards the uncertain region. This controller can also be extended for obstacle avoidance.
  • Reinforcement Learning Policy: Within the uncertain region $\hat{S}_u$, a locally learned RL policy takes over. This policy is trained using raw sensory inputs (wrist camera images and robot velocities) and a sparse reward signal. Using local observations is hypothesized to improve generalization.
  • Closing the Loop: The system can also learn from successful task completions to reduce the uncertainty region $\hat{S}_u$, allowing the MB controller to be used in larger areas in subsequent trials.

Datasets

The paper uses synthetic data generated with domain randomization (using NDDS) to train the DOPE perception system. The RL policy is trained on real-world robot interactions for the peg insertion task. No specific named datasets are listed for the RL training beyond the real-world robot trials.

Key Insights and Ideas

  • Uncertainty as a Switch: The key insight is using perception uncertainty not just as a measure of error, but as an active signal to dynamically switch control strategies. This allows for a more principled and efficient integration of MB and RL.
  • Local RL for Sample Efficiency: Training the RL policy on local observations (e.g., wrist camera) rather than global ones is proposed to improve sample efficiency and generalization, as the policy only needs to learn fine-grained control within a smaller, localized area.
  • MB as a "Funnel": The MB controller acts as a "funnel," efficiently guiding the robot into the region where RL is needed, thereby significantly reducing the exploration space and sample requirements for RL.
  • Robustness to Imperfect Models: By combining these approaches, GUAPO can overcome inaccuracies in both perception and the underlying robot/environment models, which is a significant challenge in real-world robotics.

Key Results

  • Superior Performance: GUAPO achieved a 93% insertion rate after 90 minutes of training on a real robot, significantly outperforming pure RL (SAC, 0% success) and a Residual learning baseline (0% success).
  • Sample Efficiency: GUAPO demonstrated sample efficiency by achieving successful insertions within approximately 12 minutes of real-world training time (around 8 training iterations).
  • Comparison to Baselines:
    • MB-Perfect (oracle) achieved 100% success.
  • MB-Rand-Perfect (MB with random actions) also performed well (86.67% success).
  • MB-DOPE (MB with imperfect perception) dropped to 26.6% success.
  • MB-Rand-Dope (MB with imperfect perception and random actions) improved to 70% success, suggesting random actions can help offset perception errors but are not a complete solution.
  • SAC and Residual methods failed to achieve any insertions.
  • Region Entry: GUAPO consistently entered the uncertain region $\hat{S}_u$ (100%) and the goal region $S_u$ (100%), indicating effective guidance by the MB component.

Limitations and Future Work

  • Perception System Dependency: The performance of GUAPO is still dependent on the quality of the perception system and its uncertainty estimates. While it handles errors, a completely unreliable perception system would still be problematic.
  • Task Specificity: The current implementation is demonstrated on a peg insertion task. Generalizing GUAPO to a wider range of manipulation tasks would be a natural next step.
  • Defining Uncertainty Regions: The paper assumes a "rough description" of the area of interest ($S_u$) can be provided. Automating or learning this definition could further enhance the method's generality.
  • Future Work: The authors suggest that the framework can be extended to handle more complex scenarios, including obstacle avoidance, and that the learned RL policy can be used to refine the uncertainty estimates, further closing the loop.

Summary updated

Abstract

Traditional robotic approaches rely on an accurate model of the environment, a detailed description of how to perform the task, and a robust perception system to keep track of the current state. On the other hand, reinforcement learning approaches can operate directly from raw sensory inputs with only a reward signal to describe the task, but are extremely sample-inefficient and brittle. In this work, we combine the strengths of model-based methods with the flexibility of learning-based methods to obtain a general method that is able to overcome inaccuracies in the robotics perception/actuation pipeline, while requiring minimal interactions with the environment. This is achieved by leveraging uncertainty estimates to divide the space in regions where the given model-based policy is reliable, and regions where it may have flaws or not be well defined. In these uncertain regions, we show that a locally learned-policy can be used directly with raw sensory inputs. We test our algorithm, Guided Uncertainty-Aware Policy Optimization (GUAPO), on a real-world robot performing peg insertion. Videos are available at https://sites.google.com/view/guapo-rl
BibTeX
@inproceedings{lee2020guided,
  author = {Michelle A. Lee and Carlos Florensa and Jonathan Tremblay and Nathan D. Ratliff and Animesh Garg and Fabio Ramos and Dieter Fox},
  title = {{Guided Uncertainty-Aware Policy Optimization: Combining Learning and Model-Based Strategies for Sample-Efficient Policy Learning}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2020},
  doi = {10.1109/icra40945.2020.9197125},
}
Figure