Socially Aware Motion Planning with Deep Reinforcement Learning

Miao Liu, Yu Fan Chen, Michael Everett, Miao Liu 0001, Jonathan P. How

TL;DR

This paper introduces SA-CADRL, a deep reinforcement learning approach for robotic navigation that learns to mimic human social norms for passing and interacting with pedestrians. By incorporating a novel reward shaping mechanism, the system can induce specific behaviors like "passing on the right," enabling safer and more natural autonomous navigation in crowded environments.

What Problem Does This Paper Address?

Navigating safely and efficiently in environments populated by humans, such as airports or shopping malls, is a significant challenge for autonomous robots. Humans follow subtle, often unstated, social norms when interacting and moving around each other (e.g., passing on the right). While traditional methods treat pedestrians as simple dynamic obstacles, they often result in unnatural or unsafe robot behavior. More advanced methods attempt to predict pedestrian intent, but this can lead to the "freezing robot problem" where no feasible path is found. Existing approaches struggle to quantify and generalize these human social behaviors, which are inherently stochastic and vary between individuals and cultures. This paper aims to address this by developing a method that can learn and induce socially aware navigation behaviors.

Methods

The core of this work is the Socially Aware Collision Avoidance with Deep Reinforcement Learning (SA-CADRL) algorithm. It builds upon the existing Collision Avoidance with Deep Reinforcement Learning (CADRL) framework.

  • Reinforcement Learning Foundation: The problem is framed as a sequential decision-making problem where a robot agent learns a policy to minimize time to goal while avoiding collisions. This is done by learning an optimal value function that estimates the expected time to reach the goal.
  • Deep Neural Networks: Deep neural networks are used to represent the value function, allowing the system to handle high-dimensional state spaces (including positions, velocities, and goals of nearby agents).
  • Inducing Social Norms (Key Novelty): The main innovation is a reward shaping technique to induce specific social norms. Instead of trying to explicitly model complex human behaviors, the authors propose penalizing behaviors that violate a desired social norm (e.g., passing on the left when the norm is to pass on the right). This penalty is easy to specify and doesn't need to be precise, as it simply breaks the symmetry and favors the desired behavior.
  • Multiagent Generalization: To handle more than two agents, a symmetrical neural network structure with weight-sharing and max-pooling layers is introduced. This allows the network to treat all agents symmetrically, making it scalable to scenarios with many pedestrians.
  • Exploiting Symmetry: To efficiently explore different social norms (left-handed vs. right-handed), the training process involves reflecting trajectories across the x-axis with a certain probability. This helps the agent learn alternative topologies of interaction.

Datasets

The paper does not explicitly mention using pre-existing datasets for training or evaluation. Instead, it relies on:

  • Randomly Generated Test Cases: Trajectories are generated on randomly created scenarios with varying numbers of agents, initial conditions, and goals.
  • Simulated Environments: The training and evaluation are performed in simulated environments.
  • Real-World Data (for Hardware): For the hardware experiment, the vehicle uses sensor data (Lidar, Realsense, webcams) to perceive the environment and pedestrians in real-time.

Key Insights and Ideas

  • Social Norms as Emergent Behaviors: The paper posits that complex social norms might not be explicitly programmed but rather emerge from simple, time-efficient, reciprocal collision avoidance mechanisms.
  • "What Not to Do" is Easier to Specify: Directly modeling human behavior is hard due to its stochasticity. However, specifying what constitutes a violation of a social norm (e.g., cutting someone off) is much more straightforward.
  • Reward Shaping for Norm Induction: A simple penalty for violating a desired norm can effectively guide the learning agent towards exhibiting that specific behavior, without needing to explicitly define the norm's mechanics.
  • Symmetry in Multiagent Systems: Leveraging the inherent symmetry in multiagent interactions (e.g., passing left or right) can significantly improve learning efficiency and allow for the exploration of different behavioral conventions.
  • Trade-off Between Time and Social Compliance: The experiments show that strictly adhering to social norms can sometimes lead to slightly longer travel times, highlighting a natural trade-off that can be tuned.

Key Results

  • Induction of Social Norms: SA-CADRL successfully learned and exhibited distinct left-handed and right-handed passing behaviors when trained with the corresponding reward shaping.
  • Improved Social Compliance: Compared to standard CADRL and ORCA, SA-CADRL (even without explicit norm induction) showed a stronger preference for certain navigation conventions. The SA-CADRL policies trained with norm induction achieved a strong preference split (e.g., 85-15) for the desired norm.
  • Time Efficiency: The learned policies were time-efficient, with SA-CADRL policies generally producing faster paths than ORCA, though sometimes with a slight increase in travel time (around 20%) when strictly adhering to norms.
  • Hardware Demonstration: The SA-CADRL policy was successfully implemented on a robotic vehicle, enabling autonomous navigation at human walking speed in a dynamic indoor environment with pedestrians, demonstrating real-world applicability. The vehicle generally respected the right-handed norm.

Limitations and Future Work

  • Generalizability of Norms: While the paper shows induction of specific norms, the generalizability to a wider range of complex social interactions or cultural variations is not fully explored.
  • Human Interpretation vs. Learned Behavior: The paper notes that emergent behaviors from CADRL (without norm induction) might not align with human interpretation of social norms, suggesting that explicit induction is necessary for human-like behavior.
  • Group Behavior: The current work focuses on pairwise or multiagent interactions. Future work aims to consider relationships within groups of people who walk together.
  • Real-time Performance: While the system achieves real-time performance (8.7ms per query for a 4-agent network), further optimization for more complex scenarios might be needed.

Summary updated

Abstract

For robotic vehicles to navigate safely and efficiently in pedestrian-rich environments, it is important to model subtle human behaviors and navigation rules (e.g., passing on the right). However, while instinctive to humans, socially compliant navigation is still difficult to quantify due to the stochasticity in people's behaviors. Existing works are mostly focused on using feature-matching techniques to describe and imitate human paths, but often do not generalize well since the feature values can vary from person to person, and even run to run. This work notes that while it is challenging to directly specify the details of what to do (precise mechanisms of human navigation), it is straightforward to specify what not to do (violations of social norms). Specifically, using deep reinforcement learning, this work develops a time-efficient navigation policy that respects common social norms. The proposed method is shown to enable fully autonomous navigation of a robotic vehicle moving at human walking speed in an environment with many pedestrians.
BibTeX
@inproceedings{liu2017socially,
  author = {Miao Liu and Yu Fan Chen and Michael Everett and Miao Liu and Jonathan P. How},
  title = {{Socially Aware Motion Planning with Deep Reinforcement Learning}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2017},
  doi = {10.1109/iros.2017.8202312},
}
Figure