Abstract
Given access to accurate dynamical models, modern planning approaches are effective in computing feasible and optimal plans for repetitive robotic tasks. However, it is difficult to model the true dynamics of the real world before execution, especially for tasks requiring interactions with objects whose parameters are unknown. A recent planning approach, CMAX, tackles this problem by adapting the planner online during execution to bias the resulting plans away from inaccurately modeled regions. CMAX, while being provably guaranteed to reach the goal, requires strong assumptions on the accuracy of the model used for planning and fails to improve the quality of the solution over repetitions of the same task. In this paper we propose CMAX++, an approach that leverages real-world experience to improve the quality of resulting plans over successive repetitions of a robotic task. CMAX++ achieves this by integrating model-free learning using acquired experience with model-based planning using the potentially inaccurate model. We provide provable guarantees on the completeness and asymptotic convergence of CMAX++ to the optimal path cost as the number of repetitions increases. CMAX++ is also shown to outperform baselines in simulated robotic tasks including 3D mobile robot navigation where the track friction is incorrectly modeled, and a 7D pick-and-place task where the mass of the object is unknown leading to discrepancy between true and modeled dynamics.
Introduction
Robotic planning typically relies on "forward models"—simulators that predict the outcome of actions. However, these models are rarely perfect; robots often encounter unknown friction levels or unexpected object masses. When a model is inaccurate, the resulting plan often fails during execution.
A previous approach, CMAX, attempted to mitigate this by identifying regions where the model is incorrect and inflating their cost to bias the planner away from them. However, CMAX has significant limitations: it cannot handle tasks where the robot must pass through an inaccurately modeled region to reach the goal, and it fails to improve performance over repeated trials, remaining perpetually cautious.
Methodology: The CMAX++ Framework
CMAX++ treats model inaccuracies as opportunities for learning rather than just obstacles to be avoided. It employs a hybrid planning strategy:
- Hybrid Planning: For regions where the simulator is accurate, CMAX++ uses standard model-based planning. When the robot encounters a state-action pair previously flagged as "incorrect" due to execution errors, it switches to a model-free Q-value estimate derived from real-world experience.
- Adaptive Planning (A-CMAX++): To balance initial reliability with long-term efficiency, the adaptive version (A-CMAX++) begins conservatively. It initially prioritizes finding any successful path (like CMAX), but as data accumulates, an adaptive parameter ($\alpha$) shifts the behavior toward optimizing the path for speed and cost.
- Complexity Management: To scale to high-dimensional tasks, such as a 7-DOF arm, CMAX++ utilizes hyperspheres to map regions of model error and Neural Networks to approximate the value functions in those regions.
Experimental Setup
The framework was evaluated on two primary simulated benchmarks:
- 3D Mobile Robot Navigation: A robot navigates a track containing "icy patches" with unknown friction. The internal model incorrectly assumes uniform friction across the entire track.
- 7D Pick-and-Place: A PR2 robot arm must move a heavy dumbbell. The internal model assumes the object is light, causing the arm to "skid" or fail during the lift.
Key Results and Discussion
CMAX++ demonstrates superior reliability and efficiency compared to both pure model-based and pure model-free approaches. In the 3D navigation task, the original CMAX failed to complete 200 laps in 80% of instances because it became stuck in icy patches. In contrast, CMAX++ and A-CMAX++ successfully completed all laps in every instance.
In the 7D pick-and-place task, A-CMAX++ reached the goal in 11–17 steps. A standard model-free Q-learning baseline required over 172 steps for its first success, as it lacked the "head start" provided by the inaccurate model. These results highlight that CMAX++ effectively "patches" a faulty simulator with real-world experience, eventually converging to optimal performance.
Conclusion
CMAX++ provides a robust framework for robotic planning with inaccurate models, offering mathematical guarantees for completeness and asymptotic convergence. By learning the value of actions in inaccurately modeled regions rather than attempting the computationally expensive task of fixing the simulator's physics, CMAX++ enables robots to improve their performance over time in complex, real-world environments.
Summary updated