Fast trajectory planner with a reinforcement learning-based controller for robotic manipulators

Hamidreza Kasaei, Yongliang Wang, Hamidreza Kasaei 0001

TL;DR

This paper introduces a novel framework for robotic manipulators that combines vision-based path planning with a reinforcement learning (RL) controller to achieve fast, safe, and efficient trajectory generation in complex, dynamic environments. By enhancing the Proximal Policy Optimization (PPO) algorithm with action ensembles and policy feedback, the system demonstrates improved performance in obstacle avoidance and goal-reaching tasks, even in real-world scenarios.

What Problem Does This Paper Address?

Generating obstacle-free trajectories for robotic manipulators in unstructured and cluttered environments is a significant challenge. Existing motion planning methods often require substantial computational effort to generate final trajectories by solving complex kinematic or dynamic equations. Furthermore, many current approaches struggle with dynamic environments, high-precision tasks, and adapting to different manipulator platforms. This paper aims to bridge this gap by proposing a fast, systematic, and adaptive trajectory planning framework that leverages the power of model-free reinforcement learning.

Methods

The proposed framework consists of two main components:

  1. Vision-based Trajectory Planner (Task Space): This component utilizes a large-scale "Segment Anything" (FSA) model for enhanced 3D mapping of the environment. This map is then used with B-spline optimization for kinodynamic path searching, generating waypoints for safe and dynamically feasible motion planning in the task space. This approach draws inspiration from drone path planning methods to achieve real-time performance.
  2. Reinforcement Learning-based Controller (Joint Space): This component enhances the Proximal Policy Optimization (PPO) algorithm with two key innovations:
  3. Action Ensembles (AE): This technique aggregates multiple action values from different distributions (Linear, Poisson, Beta, Exponential, Weibull) to improve stability, reduce variance, and enhance uncertainty estimation. The Weibull distribution (AEW) was found to be particularly effective.
  4. Policy Feedback (PF): This mechanism directly incorporates the policy into the value function updates, allowing the critic to adapt more rapidly to policy shifts and promoting more stable learning.

The RL controller maps task-space waypoints directly to joint-space commands, eliminating the need for explicit inverse kinematics or dynamics calculations. The state representation includes joint positions, end-effector pose, target pose, error, and obstacle distances, resulting in a 25-dimensional state vector. The action is represented by desired joint velocities.

Datasets

The paper does not explicitly mention specific datasets used for training the FSA model. However, the experiments were conducted in simulated environments (Pybullet, Gazebo) and on real-world robotic platforms. The training process involved generating simulated environments with randomly placed targets and spherical obstacles.

Key Insights and Ideas

  • Model-Free RL for Trajectory Planning: The paper strongly advocates for the potential of model-free RL over model-based approaches for obstacle-free trajectory planning in joint space, highlighting its adaptability and reduced reliance on precise environment models.
  • Synergy of Task and Joint Space Planning: Combining vision-based task-space path planning with RL-based joint-space obstacle avoidance creates a robust and efficient system.
  • PPO Enhancements are Crucial: The proposed enhancements to PPO (Action Ensembles and Policy Feedback) significantly improve its performance in terms of reward acquisition, convergence speed, stability, and adaptability for robotic manipulation tasks.
  • Weibull Distribution for Action Ensembles: The Weibull distribution for action ensembles proved to be particularly effective, offering a dynamic weighting scheme that balances exploration and exploitation, and exhibits stability across hyperparameter variations.
  • Direct Task-to-Joint Mapping: Eliminating the need for explicit inverse kinematics or dynamics calculations simplifies the control pipeline and is advantageous in cluttered or kinematically singular environments.
  • Sim-to-Real Transferability: The framework demonstrates successful transfer from simulation to real-world applications, indicating its robustness and practical applicability.

Key Results

  • Improved Performance: The enhanced PPO algorithm (with AE and PF) surpassed other baselines in performance and robustness across various simulation platforms.
  • Precise and Stable Control: The RL-based controller achieved precise and stable joint-level control of a robot manipulator.
  • Real-time Planning: The fast trajectory planner facilitated real-time trajectory planning for the manipulator, even in dynamic environments.
  • Superior Trajectories: The proposed method consistently achieved the shortest trajectories at the fastest speeds in static environments compared to five other prevalent planners.
  • High Success Rate: The system achieved a significantly higher average success rate, consistently surpassing 90% even with strict error thresholds below 10 mm.
  • Accelerated Convergence: The integrated enhancements (AE and PF) led to accelerated convergence, with the proposed approach reaching fast convergence within 1.5 million timesteps, outperforming other methods in terms of training time efficiency.
  • Generalization: The method demonstrated comparable performance across various robotic platforms with different degrees of freedom (DoF), including Baxter, Franka Emika Panda, Kuka LBR4+, and ABB Yumi.

Limitations and Future Work

  • Perception Failures due to Self-Occlusion: In real-robot experiments, the fixed position of the Kinect sensor led to self-occlusion, degrading the environment map quality. Potential solutions include multi-view camera setups or occlusion-aware perception pipelines.
  • Reactive Planning Limitations with Moving Obstacles: When the end-effector gets too close to a moving obstacle, the system may struggle to respond in time, especially with unpredictable obstacle behavior. Future work could involve predictive models of obstacle motion and increased replanning frequency.
  • Goal-Change Failures due to Spatial-Temporal Constraints: In goal-changing tasks, there might be insufficient time to re-plan before execution completes due to lags between perception, planning, and actuation. Strategies like fast-switchable motion primitives and uncertainty-aware planning are suggested.
  • Future Work: The authors plan to explore learning-based replanning strategies for more adaptive responses in dynamic environments. They also aim to incorporate lightweight model-based components to improve data efficiency and motion feasibility, and integrate their approach with high-level task and motion planning (TAMP) frameworks for more complex sequential behaviors.

Summary updated

Abstract

Generating obstacle-free trajectories for robotic manipulators in unstructured and cluttered environments remains a significant challenge. Existing motion planning methods often require additional computational effort to generate the final trajectory by solving kinematic or dynamic equations. This paper highlights the strong potential of model-free reinforcement learning methods over model-based approaches for obstacle-free trajectory planning in joint space. We propose a fast trajectory planning system for manipulators that combines vision-based path planning in task space with reinforcement learning-based obstacle avoidance in joint space. We divide the framework into two key components. The first introduces an innovative vision-based trajectory planner in task space, leveraging the large-scale fast segment anything (FSA) model in conjunction with basis spline (B-spline)-optimized kinodynamic path searching. The second component enhances the proximal policy optimization (PPO) algorithm by integrating action ensembles (AE) and policy feedback (PF), which greatly improve precision and stability in goal-reaching and obstacle avoidance within the joint space. These PPO enhancements increase the algorithm's adaptability across diverse robotic tasks, ensuring consistent execution of commands from the first component by the manipulator, while also enhancing both obstacle avoidance efficiency and reaching accuracy. The experimental results demonstrate the effectiveness of PPO enhancements, as well as simulation-to-simulation (Sim-to-Sim) and simulation-to-reality (Sim-to-Real) transfer, in improving model robustness and planner efficiency in complex scenarios. These enhancements allow the robot to perform obstacle avoidance and real-time trajectory planning in obstructed environments. Project page available at: https://sites.google.com/view/ftp4rm/home
BibTeX
@article{kasaei2025fast,
  author = {Hamidreza Kasaei and Yongliang Wang and Hamidreza Kasaei},
  title = {{Fast trajectory planner with a reinforcement learning-based controller for robotic manipulators}},
  journal = {Engineering Applications of AI},
  year = {2025},
  doi = {10.1016/j.engappai.2025.112341},
}
Figure