Introduction
Autonomous robot navigation in crowded human environments requires a sophisticated balance between efficiency and social compliance. While Deep Reinforcement Learning (DRL) offers a promising path toward learning these behaviors, the process is often hindered by the complexities of reward engineering. Manually crafting reward functions to account for goal-seeking, collision avoidance, and social norms (proxemics) frequently leads to hyperparameter redundancy and unstable training. This paper introduces the Transformable Gaussian Reward Function (TGRF), a unified framework that leverages the properties of normal distributions to simplify reward design and improve navigation safety.
The Reward Engineering Challenge
In socially-aware navigation, robots must respect human personal space. Traditional DRL approaches often employ a "kitchen sink" of disparate reward functions. This leads to several critical issues:
- Hyperparameter Redundancy: Excessive parameters make models difficult to tune.
- Reward Imbalance: Miscalculations can result in either overly aggressive behavior or the "freezing robot" problem, where the agent becomes too timid to move.
- Static Limitations: Fixed formulas often fail to capture the nuances of different social contexts or object characteristics.
Methodology: The TGRF Framework
The proposed TGRF replaces multiple mathematical formulas with a single, flexible template based on the Gaussian distribution:
$$N(x; \mu, \sigma) = \frac{1}{\sqrt{2\pi}\sigma} \cdot \exp\left(-\frac{(x-\mu)^2}{2\sigma^2}\right)$$
By adjusting the weight ($w$), mean ($\mu$), and variance ($\sigma$), the TGRF can be "sculpted" into various shapes, ranging from smooth continuous gradients to sharp, discrete penalties.
This function is integrated into a navigation model that utilizes a Gumbel Social Transformer for trajectory prediction and attention mechanisms to process human-robot interactions. The TGRF specifically defines the "danger zone" penalty, creating a smooth gradient of discomfort as the robot approaches a human, which facilitates more nuanced learning.
Experimental Results
The framework was evaluated in both 2D simulations (12m x 12m with 20 humans) and real-world trials using a Turtlebot3.
Navigation Performance
In crowded simulations, the TGRF achieved a 94.1% success rate, a significant improvement over the 77.1% success rate of the baseline model. Furthermore, the Intrusion Time Ratio (ITR)—the percentage of time the robot spends within a human's personal space—dropped to as low as 3.92%.
Learning Efficiency
Models using TGRF demonstrated faster convergence, reaching peak performance around 6,000 episodes. By providing a well-balanced reward structure, TGRF prevents the robot from getting trapped in local minima where all actions are perceived as equally detrimental.
Conclusion
The Transformable Gaussian Reward Function provides a robust and intuitive solution to the challenges of reward engineering in socially-aware navigation. By unifying diverse objectives into a single, tunable mathematical form, TGRF enhances both the training speed and the social intelligence of autonomous agents. Future work will focus on applying this framework to static obstacles and high-latency real-world environments.
Summary updated