MIDAS: Multi-agent Interaction-aware Decision-making with Adaptive Strategies for Urban Autonomous Navigation

Xiaoyi Chen, Pratik Chaudhari

TL;DR

This paper introduces MIDAS, a novel reinforcement learning approach for autonomous vehicles that learns to influence the behavior of other road users, not just predict it. By using an attention mechanism and an adaptive "driver-type" parameter, MIDAS achieves safer and more efficient navigation in complex urban environments compared to existing methods.

What Problem Does This Paper Address?

Navigating urban environments is a complex challenge for autonomous vehicles (AVs) due to the unpredictable nature of other road users, especially human drivers. Traditional approaches often rely on predicting other agents' future actions, which can lead to overly conservative driving to ensure safety. This conservatism, however, can reduce efficiency and even be exploited by human drivers. The core problem is how an AV can not only react to but also proactively influence the interactions with other agents to achieve safer and more efficient driving, mimicking how human drivers often "inch forward" to signal intent.

Methods

The paper proposes MIDAS (Multi-agent Interaction-aware Decision-making with Adaptive Strategies), a reinforcement learning (RL) based method. Key aspects include:

  • Interaction-Aware Decision-Making: Unlike prediction-based methods, MIDAS learns a policy that actively influences the actions of other agents.
  • Attention Mechanism: To handle a variable number of surrounding agents, MIDAS employs a set-based attention mechanism. This allows the AV to focus on the most relevant agents for decision-making, making it scalable and efficient.
  • Adaptive "Driver-Type" Parameter: A learned parameter, "driver-type" ($\beta$), allows the AV's policy to be tuned for different driving styles, from aggressive to cautious, without retraining. This parameter is only known to the ego agent.
  • Off-Policy RL with Improvements: MIDAS utilizes off-policy reinforcement learning, incorporating novel techniques to improve training stability and reduce variance in value function estimation. These include mixing parameters from multiple Q-value heads and averaging actions for policy evaluation.
  • Simulation Environment: A custom simulation environment was developed to facilitate diverse interaction experiments across various road geometries (T-intersections, corners, roundabouts) and agent densities.

Datasets

The paper does not explicitly mention using pre-existing datasets for training. Instead, it introduces a custom simulation environment designed for evaluating multi-agent interaction scenarios. This environment includes:

  • Various road geometries (T-intersections, corners, roundabouts).
  • The ability to generate different types of episodes: generic, collision-prone, and interaction-heavy.
  • A curated dataset of 1917 interaction episodes, created by randomizing agent numbers, driver types, arrival times, and road geometries, to ensure reproducibility and a comprehensive evaluation of interaction scenarios.

Key Insights and Ideas

  • Influence is Key: The most significant insight is that an AV can achieve better outcomes by learning to influence other agents' behavior, not just predict it. This is a paradigm shift from purely reactive strategies.
  • Attention for Scalability: The use of a set-based attention mechanism is crucial for handling the dynamic and variable number of agents in urban driving, allowing the AV to focus on what matters.
  • Tunable Driving Styles: The "driver-type" parameter provides a practical way to adapt the AV's behavior to different contexts or user preferences, offering a spectrum from cautious to aggressive driving.
  • Robustness to Uncertainty: MIDAS demonstrates robustness to variations in other agents' driving policies, a critical requirement for real-world deployment.

Key Results

  • Safety and Efficiency: MIDAS significantly reduces collision rates compared to other learned models and is as efficient as a rule-based Oracle planner, achieving comparable time-to-finish metrics.
  • Generalization: The model generalizes well across different driving policies of external agents, showing minimal degradation in performance when noise is added to other agents' actions.
  • Adaptive Behavior: The "driver-type" parameter effectively tunes the AV's aggressiveness, allowing for more efficient planning as the driver-type increases, while maintaining a low collision rate.
  • Interaction Influence: MIDAS actively influences interactions, driving forward in ambiguous situations and stopping when a collision is more likely, mimicking human-like negotiation.
  • Performance Across Geometries: MIDAS performs consistently well across different road geometries (T-intersections, roundabouts).

Limitations and Future Work

  • Simulation to Reality Gap: The authors acknowledge that deep RL policies can have high variance in real-world performance and are blind to behaviors not present in training data. Translating these policies to reality remains a challenge.
  • Intent-Aware Prediction: Future work could involve building intent-aware prediction models and performing probabilistic reasoning over their outcomes to further enhance policy learning.
  • Model-Based RL Integration: While MIDAS is model-free, incorporating prediction models could narrow the focus to interactions and potentially improve efficiency.
  • Real-World Deployment: The authors suggest their work can provide a "prior" for existing planning algorithms (like RRT or MPC) to improve their performance in autonomous driving.

Summary updated

Abstract

Autonomous navigation in crowded, complex urban environments requires interacting with other agents on the road. A common solution to this problem is to use a prediction model to guess the likely future actions of other agents. While this is reasonable, it leads to overly conservative plans because it does not explicitly model the mutual influence of the actions of interacting agents. This paper builds a reinforcement learning-based method named MIDAS where an Ego agent learns to affect the control actions of other cars in urban driving scenarios. MIDAS uses an attention mechanism to handle an arbitrary number of other agents and includes a "driver-type" parameter to learn a single policy that works across different planning objectives. We build a simulation environment that enables diverse interaction experiments with a large number of agents and develop methods for quantitatively studying the safety, efficiency, and interaction among vehicles. MIDAS is validated using extensive experiments and we show that it (i) can work across different road geometries, (ii) results in an adaptive Ego policy that can be tuned easily to satisfy different performance criteria, such as aggressive or cautious driving, (iii) is robust to changes in the driving policies of external agents, and (iv) is safer and more efficient than existing approaches to interaction-aware decision-making. Code available here.
BibTeX
@inproceedings{chen2021midas,
  author = {Xiaoyi Chen and Pratik Chaudhari},
  title = {{MIDAS: Multi-agent Interaction-aware Decision-making with Adaptive Strategies for Urban Autonomous Navigation}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2021},
  url = {https://openreview.net/forum?id=mw4eoibLsOB},
}
Figure