Integrated Intention Prediction and Decision-Making with Spectrum Attention Net and Proximal Policy Optimization

Xiaofang Zhou, Chengzhen Meng, Wenru Liu, Zengqi Peng, Ming Liu, Jun Ma

TL;DR

This paper introduces a novel integrated framework for autonomous driving that simultaneously predicts surrounding vehicle intentions and makes driving decisions. By leveraging a "spectrum attention net" to analyze frequency components of vehicle trajectories and Proximal Policy Optimization (PPO) for robust decision-making, the approach significantly improves safety, efficiency, and task success rates in complex traffic scenarios.

What Problem Does This Paper Address?

Autonomous driving systems need to predict the future actions of other vehicles (surrounding vehicles or SVs) and make safe, effective driving decisions. A major challenge is the inherent coupling between these two modules: how SVs behave influences our predictions, and our decisions can, in turn, influence SV behavior. Previous approaches often treated prediction and decision-making as separate, potentially leading to suboptimal performance in dynamic environments. Furthermore, efficiently modeling this interaction while maintaining computational feasibility has been a persistent hurdle.

Methods

The core of this work is an integrated framework that jointly learns intention prediction and decision-making.

  • Spectrum Attention Net (Intention Prediction): This novel component predicts SV intentions by analyzing the frequency spectrum of their historical trajectories. It uses Short-Time Fourier Transform (STFT) to convert time-series trajectory data into a time-frequency representation (spectrogram). A multi-head spectrum attention mechanism then captures trends across different frequency components over time, allowing it to understand both long-term (LTO) and short-term (STD) intentions. This approach is computationally efficient because it predicts intentions directly, rather than decoding them into full trajectories.
  • Proximal Policy Optimization (PPO) (Decision-Making): To handle the non-stationary nature of the integrated learning process (where the prediction module's outputs change as it learns), the authors employ PPO. PPO is an reinforcement learning algorithm that uses a clipping mechanism in its objective function. This ensures that policy updates are modest and stable, preventing catastrophic performance drops when the prediction module's parameters are updated.
  • Joint Learning: The intention prediction and decision-making modules are trained together. This allows the system to learn the intricate coupling between them, leading to more effective parameter updates and overall performance.

Datasets

The paper does not explicitly mention specific datasets used for training. However, the experiments are conducted in a simulated environment using the OpenAI Gym's Highway_Env [20]. The behaviors of surrounding vehicles (SVs) are modeled using the Intelligent Driver Model (IDM) [21].

Key Insights and Ideas

  • Frequency Domain for Intention: Analyzing the frequency components of vehicle trajectories offers a powerful way to capture underlying intentions (both long-term and short-term). Different frequencies correspond to different aspects of driving behavior, providing a richer representation than time-domain data alone.
  • The Power of Joint Learning: Explicitly modeling the coupling between intention prediction and decision-making through joint training is crucial for achieving high performance in complex, interactive driving scenarios.
  • PPO for Stability: The choice of PPO is critical for managing the non-stationarity introduced by the integrated learning approach, ensuring stable and effective policy updates.
  • Efficiency through Direct Intention Prediction: Predicting intentions directly, rather than full trajectories, significantly speeds up the prediction module, contributing to the overall efficiency of the framework.

Key Results

The proposed integrated framework demonstrated superior performance compared to baseline Deep Reinforcement Learning (DRL) methods (DQN, A2C, and PPO without intention prediction) across four representative driving scenarios: Straight Road, Intersection-v0, Intersection-v1, and Roundabout.

  • Success Rate: The proposed method achieved significantly higher success rates, for example, 93% on Straight Road compared to 65% for A2C.
  • Efficiency: The framework also showed improved efficiency, with higher percentages in task completion time.
  • Safety: Safety metrics were also notably better, with the proposed method achieving higher safety percentages across all scenarios. For instance, it reached 95% safety on Straight Road, outperforming baselines.
  • Qualitative Results: Visualizations (Fig. 3) show the AV making intelligent decisions, such as changing lanes to avoid conflicts, demonstrating a balance between safety and efficiency.

Limitations and Future Work

  • Roundabout Scenario Nuance: The paper notes that while the framework performs well, it has "some difficulties in taking spatio-spectrum into account" in the roundabout scenario, which has a complex road structure. This suggests that while frequency domain analysis is powerful, integrating spatial information more effectively could further enhance performance.
  • Future Work: The authors suggest future work should focus on "taking advantage of frequency domain representation in dealing with uncertainty in autonomous driving tasks." This implies exploring how the spectral analysis can be extended to better handle unpredictable or uncertain behaviors of other agents.

Summary updated

Abstract

For autonomous driving in highly dynamic environments, it is anticipated to predict the future behaviors of surrounding vehicles (SVs) and make safe and effective decisions. However, modeling the inherent coupling effect between the prediction and decision-making modules has been a longstanding challenge, especially when there is a need to maintain appropriate computational efficiency. To tackle these problems, we propose a novel integrated intention prediction and decision-making approach, which explicitly models the coupling relationship and achieves efficient computation. Specifically, a spectrum attention net is designed to predict the intentions of SVs by capturing the trends of each frequency component over time and their interrelations. Fast computation of the intention prediction module is attained as the predicted intentions are not decoded to trajectories in the executing process. Furthermore, the proximal policy optimization (PPO) algorithm is employed to address the non-stationary problem in the framework through a modest policy update enabled by a clipping mechanism within its objective function. On the basis of these developments, the intention prediction and decision-making modules are integrated through joint learning. Experiments are conducted in representative traffic scenarios, and the results reveal that the proposed integrated framework demonstrates superior performance over several deep reinforcement learning (DRL) baselines in terms of success rate, efficiency, and safety in driving tasks.
BibTeX
@misc{zhou2024integrated,
  author = {Xiaofang Zhou and Chengzhen Meng and Wenru Liu and Zengqi Peng and Ming Liu and Jun Ma},
  title = {{Integrated Intention Prediction and Decision-Making with Spectrum Attention Net and Proximal Policy Optimization}},
  year = {2024},
  eprint = {2408.03191},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2408.03191},
}
Figure