CMAX++ : Leveraging Experience in Planning and Execution using Inaccurate Models

Anirudh Vemula, J. Andrew Bagnell, Maxim Likhachev

Abstract

Given access to accurate dynamical models, modern planning approaches are effective in computing feasible and optimal plans for repetitive robotic tasks. However, it is difficult to model the true dynamics of the real world before execution, especially for tasks requiring interactions with objects whose parameters are unknown. A recent planning approach, CMAX, tackles this problem by adapting the planner online during execution to bias the resulting plans away from inaccurately modeled regions. CMAX, while being provably guaranteed to reach the goal, requires strong assumptions on the accuracy of the model used for planning and fails to improve the quality of the solution over repetitions of the same task. In this paper we propose CMAX++, an approach that leverages real-world experience to improve the quality of resulting plans over successive repetitions of a robotic task. CMAX++ achieves this by integrating model-free learning using acquired experience with model-based planning using the potentially inaccurate model. We provide provable guarantees on the completeness and asymptotic convergence of CMAX++ to the optimal path cost as the number of repetitions increases. CMAX++ is also shown to outperform baselines in simulated robotic tasks including 3D mobile robot navigation where the track friction is incorrectly modeled, and a 7D pick-and-place task where the mass of the object is unknown leading to discrepancy between true and modeled dynamics.

Introduction

Robotic planning typically relies on a "forward model"—a mathematical description of how the environment changes in response to robot actions. However, accurately modeling the real world is notoriously difficult. A robot might misjudge the mass of a heavy object or the friction of an icy floor, leading to plan failure during execution.

Traditional model-based planners fail when the underlying physics model is wrong. A previous approach, CMAX, addressed this by identifying "incorrect" transitions and biasing the robot to avoid them. However, CMAX suffers from two major limitations:

  1. Inevitability: If the only path to a goal requires traversing an inaccurately modeled region (e.g., lifting a heavy object), CMAX fails because it attempts to avoid that region indefinitely.
  2. Suboptimality: It does not improve performance over time, merely seeking a "safe" path rather than an optimal one.

CMAX++ bridges this gap by integrating model-free reinforcement learning with model-based planning, allowing robots to learn from experience and utilize inaccurately modeled regions to find optimal paths over repeated task executions.

Methodology

The core of CMAX++ is a hybrid approach that utilizes the provided model where it is accurate and switches to model-free learning where it is not.

The planner performs a lookahead search. When it encounters a transition known to be "incorrect" based on past experience, it stops expanding that branch. Instead, it substitutes the model-based prediction with a learned model-free Q-value, which estimates the cost-to-goal from that state. For "correct" transitions, the standard model is used.

CMAX++ and Adaptive CMAX++ (A-CMAX++)

CMAX++ maintains a set of discovered incorrect transitions. If the robot's actual state during execution deviates from the model's prediction, the transition is flagged, and Q-values are updated using real-world experience.

To balance safety and optimality, Adaptive CMAX++ (A-CMAX++) was developed. It switches between a conservative strategy (avoiding errors like CMAX) and an optimistic strategy (exploiting experience like CMAX++). Early in the learning process, it prioritizes reaching the goal safely; as experience grows, it shifts toward the optimal path.

Experimental domains for CMAX++: a 7D pick-and-place task (left) and a 3D mobile robot navigation task (right).

Experimental Evaluation

The researchers evaluated CMAX++ on two simulated tasks:

  1. 3D Mobile Robot Navigation: A car-like robot navigating a track with hidden "icy patches" (unknown friction). The robot must learn to navigate through or around these patches over 200 laps.
  2. 7D Pick-and-Place: A robotic arm lifting a heavy dumbbell (modeled as light) and placing it on a high table. This involves a massive state space ($10^7$ states), requiring function approximation (Neural Networks or K-Nearest Neighbors) and "hyperspheres" to represent regions of model error.

Key Results

CMAX++ and its adaptive variant demonstrated significant improvements in both reliability and efficiency.

  • Reliability: In the navigation task, CMAX++ and A-CMAX++ successfully completed all 200 laps in 10/10 test instances. In contrast, the original CMAX failed in 80% of instances because it became "stuck" trying to avoid necessary but inaccurately modeled icy patches.
  • Efficiency: A-CMAX++ outperformed both CMAX and pure CMAX++. As shown in Figure 4, it achieved goal-driven behavior early on and converged toward the optimal path as it gained experience.
  • Scaling to High Dimensions: In the 7D arm task, A-CMAX++ reached the goal in 10-11 steps after a few repetitions. This significantly outperformed pure model-free Q-learning, which initially took over 170 steps and struggled to converge.
Average number of steps taken to finish a lap in the 3D mobile robot navigation experiment across 200 laps. The numbers above the bars indicate the number of successful trials out of 10.

Discussion and Future Work

CMAX++ demonstrates that robots do not necessarily need to "fix" a complex physics model to perform optimally. Instead, they can learn the value of actions in "broken" regions. However, the approach is sensitive to hyperparameters like the error hypersphere radius ($\delta$). Additionally, the "Optimistic Model Assumption"—that the initial model underestimates true costs—is a key requirement for guaranteed exploration. Future work may involve interleaving model identification with CMAX++ to further accelerate learning.

Summary updated

Abstract

Given access to accurate dynamical models, modern planning approaches are effective in computing feasible and optimal plans for repetitive robotic tasks. However, it is difficult to model the true dynamics of the real world before execution, especially for tasks requiring interactions with objects whose parameters are unknown. A recent planning approach, CMAX, tackles this problem by adapting the planner online during execution to bias the resulting plans away from inaccurately modeled regions. CMAX, while being provably guaranteed to reach the goal, requires strong assumptions on the accuracy of the model used for planning and fails to improve the quality of the solution over repetitions of the same task. In this paper we propose CMAX++, an approach that leverages real-world experience to improve the quality of resulting plans over successive repetitions of a robotic task. CMAX++ achieves this by integrating model-free learning using acquired experience with model-based planning using the potentially inaccurate model. We provide provable guarantees on the completeness and asymptotic convergence of CMAX++ to the optimal path cost as the number of repetitions increases. CMAX++ is also shown to outperform baselines in simulated robotic tasks including 3D mobile robot navigation where the track friction is incorrectly modeled, and a 7D pick-and-place task where the mass of the object is unknown leading to discrepancy between true and modeled dynamics.
BibTeX
@inproceedings{vemula2021cmax,
  author = {Anirudh Vemula and J. Andrew Bagnell and Maxim Likhachev},
  title = {{CMAX++ : Leveraging Experience in Planning and Execution using Inaccurate Models}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  url = {https://openreview.net/forum?id=HKUWO2u7Sxc},
}

Loading related content...

Figure