Mapless Navigation among Dynamics with Social-safety-awareness: a reinforcement learning approach from 2D laser scans

Jun Jin, Nhat M. Nguyen, Nazmus Sakib, Daniel Graves, Hengshuai Yao, Martin Jägersand

TL;DR

This paper introduces a novel reinforcement learning approach for robots to navigate unknown environments with both static obstacles and dynamic humans, using only 2D laser scans. It achieves this by incorporating "ego-safety" (avoiding self-collision) and "social-safety" (avoiding negatively impacting humans) into its decision-making, leading to more cooperative and efficient navigation.

What Problem Does This Paper Address?

Navigating autonomously in environments where the map is unknown (mapless navigation) is a significant challenge, especially when dynamic elements like humans are present. Traditional approaches often struggle with predicting human behavior, leading to the "freezing robot problem" where the robot becomes paralyzed to avoid potential collisions. Existing methods often rely on complex sensors or explicit pedestrian detection. This paper tackles the problem of achieving safe and efficient navigation using only 2D laser scans, a more common and less resource-intensive sensor, while explicitly considering the robot's impact on human safety and comfort.

Methods

The core of the proposed method is a reinforcement learning policy trained to optimize a reward function that balances three components:

  • Ego-safety: Preventing the robot from colliding with any obstacles (static or dynamic) from its own perspective. This is defined by a safety zone around the robot.
  • Social-safety: Measuring the robot's impact on surrounding pedestrians. This is quantified by predicting the intrusion of the robot's action into a pedestrian's "interaction area" (a defined zone around them based on their movement).
  • Reaching the target: A reward for making progress towards the desired destination.

The method uses a deep neural network architecture (illustrated in Figure 4) that processes historical 2D laser scan data (referred to as "motion features") and the robot's current state. A key innovation is the way motion features are processed to disentangle the robot's own movement from the movement of surrounding objects, making the policy more robust. The policy is trained using the Deep Deterministic Policy Gradient (DDPG) algorithm.

Datasets

The paper does not explicitly mention using pre-existing public datasets for training. Instead, they developed their own simulator that generates randomized environments with:

  • Randomized maps and static obstacles.
  • Dynamic pedestrians with randomized numbers, sizes, shapes, locations, and behaviors (including stop-and-go, walk-ins, and interactions with each other).

The learned policy is then evaluated in this simulator, as well as in the Gazebo simulation environment and on a real Jackal robot.

Key Insights and Ideas

  • Social-safety as a distinct metric: Explicitly defining and rewarding "social-safety" (how the robot's actions affect humans) is crucial for cooperative navigation, going beyond simple collision avoidance.
  • 2D laser scans are sufficient: The research demonstrates that even with the limitations of 2D laser scans, it's possible to achieve sophisticated navigation behaviors, including human-aware navigation.
  • Disentangling motion: Effectively separating the robot's own motion from the motion of its surroundings in the input data is key to learning robust policies.
  • Cooperative behavior emerges: The learned policies, especially the "social policy," exhibit cooperative behaviors, actively integrating into crowd flows rather than passively waiting, which leads to more efficient navigation.
  • Transferability: The policy trained in the custom simulator shows good generalization to different scenarios and environments (Gazebo, real robot) without fine-tuning.

Key Results

  • Simulator Performance: In the simulator, the proposed "social policy" achieved a 100% success rate in mapless navigation and navigation among dynamic crowds, with significantly faster arrival times (e.g., 3.1s vs. 7.0s for greedy baseline in mapless navigation).
  • Gazebo Performance: In Gazebo, the social policy also demonstrated high success rates (e.g., 100% in mapless path planning across various complexities) and often outperformed baselines in terms of efficiency and social awareness.
  • Real-world Qualitative Results: The robot demonstrated effective navigation in real-world tests, even with imperfect localization, showing agile avoidance of dynamic obstacles (like humans pushing boxes).
  • Social Score: The "social policy" consistently achieved higher "social scores" compared to the "ego policy" and baselines, indicating better awareness of human impact.

Limitations and Future Work

  • Localization Dependency: The current mapless navigation relies on a third-party localization module, which can be unrealistic in unknown environments. The authors suggest that a high-level path planner with vision sensor support could address this.
  • Training Tedium: Fine-tuning the parameters for training the reinforcement learning policy is noted as being tedious. They mention that multi-stage training strategies exist but lack theoretical guarantees.
  • Stop Behaviors: The learned policy sometimes exhibits circling behavior when trying to escape complex situations, which is not the most efficient way to stop. Learning better stop behaviors without sacrificing efficiency is a future direction.
  • Theoretical Guarantees: The paper acknowledges the lack of theoretical guarantees for the multi-stage training strategy used.

Summary updated

Abstract

We consider the problem of mapless collision-avoidance navigation where humans are present using 2D laser scans. Our proposed method uses ego-safety to measure collision from the robot's perspective and social-safety to measure the impact of robot's actions on surrounding pedestrians. Specifically, the social-safety part predicts the intrusion impact of the robot's action into the interaction area with surrounding humans. We train the policy using reinforcement learning on a simple simulator and directly evaluate the learned policy in Gazebo and real robot tests. Experiments show the learned policy smoothly transferred to different scenarios without any fine tuning. We observe that our method demonstrates time-efficient path planning behavior with high success rate in the mapless navigation task. Furthermore, we test our method in a navigation task among dynamic crowds, considering both low and high volume traffic. Our learned policy demonstrates cooperative behavior that actively drives our robot into traffic flows while showing respect to nearby pedestrians. Evaluation videos are at https://sites.google.com/view/ssw-batman.
BibTeX
@inproceedings{jin2020mapless,
  author = {Jun Jin and Nhat M. Nguyen and Nazmus Sakib and Daniel Graves and Hengshuai Yao and Martin Jägersand},
  title = {{Mapless Navigation among Dynamics with Social-safety-awareness: a reinforcement learning approach from 2D laser scans}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2020},
  url = {https://openreview.net/forum?id=VTYjPj1X8aw},
}
Figure