Transformable Gaussian Reward Function for Socially-Aware Navigation with Deep Reinforcement Learning

Jinyeob Kim, Sumin Kang, Sungwoo Yang, Beomjoon Kim, Jargalbaatar Yura, Donghan KimarXiv 2024 · DOI · arXiv

Introduction

Autonomous robot navigation in crowded human environments requires a sophisticated balance between efficiency and social compliance. While Deep Reinforcement Learning (DRL) offers a promising path toward learning these behaviors, the process is often hindered by the complexities of reward engineering. Manually crafting reward functions to account for goal-seeking, collision avoidance, and social norms (proxemics) frequently leads to hyperparameter redundancy and unstable training. This paper introduces the Transformable Gaussian Reward Function (TGRF), a unified framework that leverages the properties of normal distributions to simplify reward design and improve navigation safety.

The Reward Engineering Challenge

In socially-aware navigation, robots must respect human personal space. Traditional DRL approaches often employ a "kitchen sink" of disparate reward functions. This leads to several critical issues:

  1. Hyperparameter Redundancy: Excessive parameters make models difficult to tune.
  2. Reward Imbalance: Miscalculations can result in either overly aggressive behavior or the "freezing robot" problem, where the agent becomes too timid to move.
  3. Static Limitations: Fixed formulas often fail to capture the nuances of different social contexts or object characteristics.

Methodology: The TGRF Framework

The proposed TGRF replaces multiple mathematical formulas with a single, flexible template based on the Gaussian distribution:

$$N(x; \mu, \sigma) = \frac{1}{\sqrt{2\pi}\sigma} \cdot \exp\left(-\frac{(x-\mu)^2}{2\sigma^2}\right)$$

By adjusting the weight ($w$), mean ($\mu$), and variance ($\sigma$), the TGRF can be "sculpted" into various shapes, ranging from smooth continuous gradients to sharp, discrete penalties.

The TGRF can be transformed by adjusting the variance parameter (sigma), allowing it to represent different levels of social discomfort or physical boundaries.

This function is integrated into a navigation model that utilizes a Gumbel Social Transformer for trajectory prediction and attention mechanisms to process human-robot interactions. The TGRF specifically defines the "danger zone" penalty, creating a smooth gradient of discomfort as the robot approaches a human, which facilitates more nuanced learning.

Experimental Results

The framework was evaluated in both 2D simulations (12m x 12m with 20 humans) and real-world trials using a Turtlebot3.

In crowded simulations, the TGRF achieved a 94.1% success rate, a significant improvement over the 77.1% success rate of the baseline model. Furthermore, the Intrusion Time Ratio (ITR)—the percentage of time the robot spends within a human's personal space—dropped to as low as 3.92%.

Comparison of robot behavior: (a) shows a reactive maneuver that may lead to social intrusions, while (b) demonstrates proactive positioning enabled by the TGRF to maintain social distance.

Learning Efficiency

Models using TGRF demonstrated faster convergence, reaching peak performance around 6,000 episodes. By providing a well-balanced reward structure, TGRF prevents the robot from getting trapped in local minima where all actions are perceived as equally detrimental.

Conclusion

The Transformable Gaussian Reward Function provides a robust and intuitive solution to the challenges of reward engineering in socially-aware navigation. By unifying diverse objectives into a single, tunable mathematical form, TGRF enhances both the training speed and the social intelligence of autonomous agents. Future work will focus on applying this framework to static obstacles and high-latency real-world environments.

Summary updated

BibTeX
@misc{kim2024transformable,
  author = {Jinyeob Kim and Sumin Kang and Sungwoo Yang and Beomjoon Kim and Jargalbaatar Yura and Donghan Kim},
  title = {{Transformable Gaussian Reward Function for Socially-Aware Navigation with Deep Reinforcement Learning}},
  year = {2024},
  eprint = {2402.14569},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2402.14569},
}
Figure