Deep Reinforcement Learning With Multiple Unrelated Rewards for AGV Mapless Navigation

Boliang Cai, Changyun Wei, Ze Ji

In the world of industrial automation, Automated Guided Vehicles (AGVs) are the workhorses of the modern warehouse. Traditionally, these robots rely on pre-built maps and centralized control. However, as environments become more dynamic—filled with unpredictable humans and other robots—researchers are turning to Mapless Navigation powered by Deep Reinforcement Learning (DRL).

A new paper by Cai et al., titled "Deep Reinforcement Learning With Multiple Unrelated Rewards for AGV Mapless Navigation," introduces a clever way to solve one of DRL’s biggest headaches: the "reward shaping" problem.

TL;DR

This paper proposes Multi-Feature Policy Gradients (MFPG), a DRL framework that treats navigation and collision avoidance as two separate, unrelated reward signals. By using dual Q-networks and an adaptive weighting mechanism, the system achieves higher success rates and smoother "socially acceptable" behavior in complex, mapless environments compared to standard single-reward algorithms.

What Problem Does This Paper Address?

Reinforcement Learning agents learn by maximizing a "reward." In robot navigation, we usually want two things: reach the goal quickly and don't hit anything.

The standard approach is to mash these together into a single number (e.g., Reward = 0.7 * Navigation + 0.3 * Collision_Avoidance). This is problematic because:

  1. Hand-tuning is hard: Finding the "perfect" weights is a trial-and-error nightmare.
  2. Conflicting Gradients: A single scalar reward can mask dangerous situations. If the reward for moving toward the goal is high enough, the agent might "decide" that a minor collision is a price worth paying.
  3. Rigidity: Fixed weights don't adapt to different phases of a mission (e.g., an open hallway vs. a crowded doorway).

Methods

The authors introduce Multi-Feature Policy Gradients (MFPG) and its adaptive version, Ada-MFPG.

  • Dual Critic Networks: Instead of one "Critic" network evaluating a combined reward, MFPG uses two separate Q-networks. One evaluates the distance-shortening reward ($r_{Nav}$), and the other evaluates collision avoidance ($r_{CA}$).
  • Adaptive Weighting (Ada-MFPG): This is the "secret sauce." The algorithm automatically adjusts the weights of the two policy gradients based on the standard deviation of the state-action values. If one critic is "shouting" louder (has higher variance in its predictions), the system adjusts to balance the influence of both goals.
  • Social Norm Biasing: To make the robots predictable to humans, the authors added a continuous bias to the collision reward based on laser sensor indices. This encourages the robot to follow the "right-hand rule" when passing others.
  • Modified Prioritized Experience Replay (PER): They updated the standard PER algorithm to prioritize training samples based on the combined loss of both critic networks, ensuring the agent learns from its most complex failures.

Datasets

The researchers evaluated their approach in several high-fidelity ROS/Gazebo simulation environments:

  • Obstacle-free Square Cage: For baseline collision avoidance between four robots.
  • Large-scale Warehouse: A complex environment with static obstacles and multiple robots.
  • Unstructured Complex Environment: Featuring narrow gaps and high-density obstacles.
  • Real-world Testing: The model was deployed on physical TurtleBot 3 robots in a corridor environment to test "Sim-to-Real" performance.

Key Insights and Ideas

  • Decoupling works: By treating rewards as "unrelated," the agent avoids the "averaging out" effect of a single scalar reward. It allows the policy to respect the distinct geometry of the safety task versus the navigation task.
  • The "Aha" Moment on Weighting: Using the standard deviation of Q-values as a balancing factor is a robust alternative to manual tuning. It essentially lets the neural networks tell the controller which sub-task is currently more "uncertain" or critical.
  • Socially Aware AI: The "right-hand rule" isn't a hard-coded constraint but an emergent behavior triggered by a biased reward. This makes the robot's pathing more intuitive for human coworkers.

Key Results

  • Higher Success Rates: In warehouse simulations, MFPG achieved an 80.5% success rate, significantly outperforming standard DDPG (73.2%).
  • Efficiency: MFPG-based algorithms showed the optimal task completion duration and lower variance in pathing compared to baselines like SAC (Soft Actor-Critic) and TD3 (Twin Delayed DDPG).
  • Safety: The "Crash Distance" (how close the robot gets to obstacles before reacting) was significantly improved, with MFPG maintaining a much more stable safety margin.
  • Real-World Viability: The model successfully transferred to physical robots, navigating narrow doors and avoiding dynamic obstacles (humans) in real-time at a 5Hz update rate.

Limitations and Future Work

While the results are promising, the authors acknowledge a few hurdles:

  • Localization Dependency: The system still relies on reasonably accurate localization (odometry/SLAM). On slippery warehouse floors, odometry errors can lead to "drifting" that the current DRL policy struggles to correct.
  • Sensor Uncertainty: The current model assumes "clean" laser data. Future work will look at incorporating sensor noise and uncertainty directly into the state representation.
  • Stability: While the convergence was proved mathematically, the stability of the dual-critic system in extremely high-density multi-agent scenarios (e.g., 20+ robots) remains to be tested.

Source Code: The authors have made their implementation available on GitHub at dornenkrone/MFPG.

Summary updated

Abstract

Mapless navigation for Automated Guided Vehicles (AGV) via Deep Reinforcement Learning (DRL) algorithms has attracted significantly rising attention in recent years. Collision avoidance from dynamic obstacles in unstructured environments, such as pedestrians and other vehicles, is one of the key challenges for mapless navigation. Autonomous navigation requires a policy to make decisions to optimize the path distance towards the goal but also to reduce the probability of collisions with obstacles. Mostly, the reward for AGV navigation is calculated by combining multiple reward functions for different purposes, such as encouraging the robot to move towards the goal or avoiding collisions, as a state-conditioned function. The combined reward, however, may lead to biased behaviours due to the empirically chosen weights when multiple rewards are combined and dangerous situations are misjudged. Therefore, this paper proposes a learning-based method with multiple unrelated rewards, which represent the evaluation of different behaviours respectively. The policy network, named Multi-Feature Policy Gradients (MFPG), is conducted by two separate Q networks that are constructed by two individual rewards, corresponding to goal distance shortening and collision avoidance, respectively. In addition, we also propose an auto-tuning method, named Ada-MFPG, that allows the MFPG algorithm to automatically adjust the weights for the two separate policy gradients. For collision avoidance, we present a new social norm-oriented continuous biased reward for performing specific social norm so as to reduce the probabilities of AGV collisions. By adding an offset gain to one of the reward functions, vehicles conducted by the proposed algorithm exhibited the predetermined features. The work was tested in different simulation environments under multiple scenarios with a single robot or multiple robots. The proposed MFPG method is compared with standard Deep Deterministic Policy Gradient (DDPG), the modified DDPG, SAC and TD3 with a social norm mechanism. MFPG significantly increases the success rate in robot navigation tasks compared with the DDPG. Besides, among all the benchmarking algorithms, the MFPG-based algorithms have the optimal task completion duration and lower variance compared with the baselines. The work has also been tested on real robots. Experiments on the real robots demonstrate the viability of the trained model for the real world scenarios. The learned model can be used for multi-robot mapless navigation in complex environments, such as a warehouse, that need multi-robot cooperation. Our source code and supplementary material is available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/dornenkrone/MFPG</uri> Note to Practitioners—Autonomous navigation for AGVs in complex and large-scale environments, such as factories and warehouses, is challenging. AGVs are usually centrally controlled and depend on reliable communications. However, centralized control is not always reliable due to poor signal strengths or crashes of the server, and hence unsuitable due to the requirements of accurate information of the dynamic environments and fast responses of decision making. Therefore, it is necessary for the vehicles to perform reliable decision making based on only onboard sensors and processors, for efficient and safe autonomous navigation. Existing methods, such as simultaneous localization and mapping (SLAM) and motion planning algorithms, have been widely used. However, they are neither flexible nor generalizable enough. This paper proposes a method for autonomous navigation based on reinforcement learning (RL), which allows vehicles to gain experience through cumulative rewards by continuously interacting with the environment. The RL-based controller is designed for optimising its performance in two independent aspects, namely collision avoidance and navigation, which are quantified as separate rewards. Instead of carefully hand-crafting a combined reward, our proposed approach trains the agent using the two rewards separately to obtain one optimal policy. It is clearly easier and more practical to design the individual rewards than manually combining them. Besides, the algorithm includes a mechanism for incorporating social norms to encourage the vehicles to follow the right-hand rule, such that they can avoid pedestrians or other vehicles in a socially acceptable manner. This is achieved by adding a continuous bias on the collision avoidance reward. Experiments using simulation environments and real robots suggest that the method is generalizable to multi-robot systems, while guaranteeing safety. In future research, we will focus on incorporating uncertainties of sensor readings for safe and reliable autonomous navigation.
BibTeX
@article{cai2024deep,
  author = {Boliang Cai and Changyun Wei and Ze Ji},
  title = {{Deep Reinforcement Learning With Multiple Unrelated Rewards for AGV Mapless Navigation}},
  journal = {IEEE Transactions on Automation Science and Engineering},
  year = {2024},
  doi = {10.1109/tase.2024.3410162},
}
Figure