Abstract
Mapless navigation for mobile Unmanned Ground Vehicles (UGVs) using Deep Reinforcement Learning (DRL) has attracted significantly rising attention in robotic and related research communities. Collision avoidance from dynamic obstacles in unstructured environments, such as pedestrians and other vehicles, is one of the key challenges for mapless navigation. This paper proposes a DRL algorithm based on heuristic correction learning for autonomous navigation of a UGV in mapless configuration. We use a 24-dimensional lidar sensor, and merge the target position information and the speed information of the UGV as the input of the reinforcement learning agent. The actions of the UGV are produced as the output of the agent. Our proposed algorithm has been trained and evaluated in both static and dynamic environments. The experimental result shows that our proposed algorithm can reach the target in less time with shorter distances under the premise of ensuring safety than other algorithms. Especially, the success rate of our proposed algorithm is 2.05 times higher than the second effective algorithm and the trajectory efficiency is improved by 24% in the dynamic environment. Finally, our proposed algorithm is deployed on a real robot in the real-world environment to validate and evaluate the algorithm performance. Experimental results show that our proposed algorithm can be directly applied to real robots robustly.
Introduction
Navigating an Unmanned Ground Vehicle (UGV) through crowded, mapless environments—such as busy hospital corridors or smart factories—is a major challenge in robotics. Traditional methods often rely on pre-built global maps (SLAM), which can be expensive to maintain and prone to failure in dynamic settings. Deep Reinforcement Learning (DRL) offers a mapless alternative where robots learn to navigate via trial and error. However, standard DRL faces hurdles including slow convergence, safety risks during exploration, and the "sim-to-real" gap. This paper introduces TD3-HCP, a framework that addresses these issues by hybridizing DRL with classical heuristic-based safety mechanisms.
Methodology
The proposed framework integrates Twin Delayed Deep Deterministic Policy Gradient (TD3) with a Heuristic Correction Policy (HCP) to ensure safe and efficient navigation.
System Architecture
The system processes environmental data through a state-actor-new state loop. The input state consists of 24-dimensional lidar readings, relative target position, and current UGV velocity.
TD3 and Heuristic Correction Policy (HCP)
TD3 is employed as the base algorithm to mitigate overestimation bias through dual critic networks. The core innovation is the HCP, which acts as a safety mechanism. When the robot enters a high-risk state (determined by lidar data), an Artificial Potential Field (APF) calculates a corrective steering angle.
Instead of a simple override, the system uses a competitive mechanism: both the DRL-suggested action and the HCP-suggested action are evaluated by the Critic network. The action with the higher predicted Q-value is executed. This allows the agent to effectively "learn" from the heuristic's expertise during training, leading to faster and more stable convergence.
Improved Prioritized Experience Replay (PER)
The authors enhanced standard PER by incorporating the loss of the actor network into the transition priority calculation. This focuses training on scenarios where the robot's movement policy is most uncertain, further accelerating the learning process.
Experimental Results
The model was evaluated in Gazebo simulations (static and dynamic) and on a physical Husky robot.
- Dynamic Environments: In a 24x24m area with 12 moving pedestrians, TD3-HCP achieved a 74% success rate, significantly outperforming standard TD3 (36%) and DDPG-PER (18%).
- Efficiency: Trajectory efficiency improved by 24%, indicating more direct paths to goals.
- Real-World Performance: The policy transferred successfully to a physical UGV, which demonstrated real-time obstacle avoidance and goal-reaching in an indoor corridor, circumscribing a pedestrian within a 12-second window.
Conclusion
TD3-HCP demonstrates that combining "black-box" AI with classical geometric heuristics provides a robust solution for mapless UGV navigation. By using heuristics as a teacher rather than just a fallback, the agent achieves superior safety and efficiency in complex, dynamic environments. Future work will explore multi-agent coordination and 3D sensor integration.
Summary updated