Deep reinforcement learning-based composite path planning with key path points

Chao Sun, Jiang Wang, Chaoxu Mu, Lei Xing, Zhenguo SunScientia 2025 · DOI

Abstract

This paper proposes a robot path planning method that combines Voronoi diagrams with deep reinforcement learning (DRL). In the global planning phase, a "safe zone" is constructed via Voronoi diagrams, and A along with key path point extraction algorithms are used to optimize path points, providing strategic targets for local planning. Local planning utilizes a novel SW-RDQN* (Stage-key & Weight-balanced Replay Rainbow DQN) algorithm, which enhances adaptability to complex dynamic environments through multi-stage path segmentation, trajectory prediction-based rewards, and weight-balanced replay training. By fusing convolutional and multi-layer perceptron features with an LSTM model, the system improves the robot's perception and response in dynamic settings. This approach effectively increases path planning accuracy and robustness in dynamic environments.

Introduction: The Challenge of Dynamic Navigation

Robot path planning is traditionally divided into two distinct domains: global planning and local planning. Global algorithms like A* or Dijkstra are excellent at finding the shortest path on static maps but fail to account for moving obstacles. Conversely, local planning methods based on Deep Reinforcement Learning excel at reacting to dynamic objects but often suffer from "sparse rewards"—where the robot lacks feedback until it reaches a final goal—and can easily become trapped in local optima.

This research bridges the gap between these two approaches by presenting a composite framework that uses geometric skeletons for global strategy and specialized DRL for local tactics.

The Composite Framework

The proposed system operates in two main stages, as illustrated in the overall architecture below. The global planner sets the "strategy" by identifying safe corridors, while the local planner handles the "tactics" of avoiding unpredictable moving obstacles.

The composite path planning framework, illustrating the transition from global strategy to local execution.

Global Strategy: Voronoi Skeletons and Key Points

Instead of a standard grid-based search, the global planner constructs a Voronoi diagram. This geometric approach identifies paths that maximize the distance from all known static obstacles, naturally biasing the robot toward the safest possible routes (e.g., the center of a hallway).

To make this global path useful for a DRL agent, the system extracts "Key Path Points." After finding a path through the Voronoi skeleton using A*, a pruning algorithm removes redundant nodes. These key points serve as intermediate sub-goals, effectively breaking down a long-range navigation task into a series of manageable stages.

The detailed process of extracting the Voronoi skeleton and identifying key path points to guide the local planner.

Local Tactics: SW-RDQN

The local planner utilizes a modified "Rainbow" DQN architecture, dubbed SW-RDQN, featuring several key innovations:

  1. Stage-key Mechanism: The robot is rewarded for reaching the intermediate key points provided by the global planner. This solves the sparse reward problem by providing frequent positive reinforcement.
  2. Trajectory Prediction (PLP): The state input includes a "Predicted Landing Point" for dynamic obstacles, calculated via a Recurrent Neural Network (RNN). This allows the robot to anticipate where an obstacle will be, rather than just reacting to where it is currently.
  3. Weight-balanced Replay: Standard DRL uses Prioritized Experience Replay based on temporal difference (TD) error. This paper introduces a reward-based weight, ensuring the agent learns equally from its most successful maneuvers and its critical failures.
  4. Temporal Awareness: The architecture uses a multi-modal input (CNN for visual data, MLP for numerical data) feeding into an LSTM layer. This gives the robot a "memory" of past frames, allowing it to understand the velocity and intent of moving obstacles.

Key Results and Performance

The framework was evaluated in complex simulated environments (Interior, Open, and Maze scenes) featuring up to 30 static and 10 dynamic obstacles.

  • Success Rate: In high-density dynamic environments, the proposed method achieved a 76.1% success rate, significantly outperforming vanilla DQN (5%) and advanced methods like SAC or TD3.
  • Efficiency: The system proved highly efficient with an average execution time of 291.0 ms, compared to 851.3 ms for Improved Particle Swarm Optimization (IPSO).
  • Path Quality: The average path length was 349.15 units, nearly 50% more efficient than the IPSO baseline of 663.2 units.
  • Ablation Insights: Removing the trajectory prediction module caused the success rate to drop from ~86% to 65%, highlighting the critical importance of predictive modeling in dynamic safety.

Conclusion

By combining the geometric safety of Voronoi diagrams with the adaptive power of SW-RDQN, this composite framework offers a robust solution for robot navigation in unpredictable environments. Future work will look toward transitioning from discrete to continuous action spaces to improve movement smoothness and exploring multi-agent cooperation in shared spaces.

Summary updated

BibTeX
@inproceedings{sun2025deep,
  author = {Chao Sun and Jiang Wang and Chaoxu Mu and Lei Xing and Zhenguo Sun},
  title = {{Deep reinforcement learning-based composite path planning with key path points}},
  booktitle = {Scientia},
  year = {2025},
  doi = {10.1360/ssi-2025-0101},
}
Figure