Transferable Deep Reinforcement Learning Framework for Autonomous Vehicles With Joint Radar-Data Communications

Nguyen Quang Hieu, Dinh Thai Hoang, Dusit Niyato, Ping Wang, Dong In Kim, Chau Yuen

TL;DR

This paper introduces a framework that helps autonomous vehicles (AVs) balance two critical but competing tasks: radar sensing for safety and data communication for coordination. By combining Double Deep Q-Networks (DDQN) with a novel Transfer Learning mechanism (TLwD), the authors enable AVs to make optimal decisions in real-time and adapt to new driving environments up to 67% more safely and significantly faster than traditional reinforcement learning methods.

What Problem Does This Paper Address?

Modern AVs rely on Joint Radar-Communications (JRC) systems, where radar and data transmission share the same hardware and frequency bands (like the 71–76 GHz mm-wave band). This saves space and power but creates a zero-sum game: every millisecond spent transmitting data is a millisecond not spent scanning for obstacles.

The challenge is that the "right" balance changes constantly. In heavy rain or high-speed traffic, missing a radar scan is dangerous. In clear weather on a quiet road, the vehicle should prioritize uploading sensor data to the cloud. Furthermore, when an AV moves from a quiet rural road to a chaotic urban center, a pre-trained AI often fails because the "rules" of the environment have shifted. The authors tackle how to optimize these decisions dynamically and how to "transfer" knowledge so an AV doesn't have to relearn how to drive every time it enters a new city.

Methods

The researchers modeled the AV’s decision-making process as a Markov Decision Process (MDP). To solve it, they employed two primary AI techniques:

  1. Double Deep Q-Network (DDQN): Unlike standard Q-learning, which can over-optimistically estimate the value of certain actions, DDQN uses two separate neural networks to decouple action selection from evaluation. This leads to much more stable and reliable learning in complex environments.
  2. Transfer Learning with Demonstrations (TLwD): This is the paper's core innovation. Instead of starting from scratch in a new environment, the AV uses a small set of "demonstration data" from a previous environment. It uses Prioritized Experience Replay, which forces the AI to "study" the most important or surprising past experiences more frequently, and multi-step Temporal Difference (TD) learning to look further ahead into the future when calculating rewards.

The system chooses between multiple radar modes (long-range vs. high-resolution short-range) and various data transmission rates based on five environmental factors: channel state, road condition, weather, vehicle speed, and nearby traffic density.

Datasets

To ensure the simulations were grounded in reality, the authors used the nuScenes dataset.

  • nuScenes: A massive public dataset containing 242km of driving data from Boston and Singapore.
  • Application: The researchers extracted real-world distributions for traffic density, car dimensions, and pedestrian sizes. This allowed them to simulate "Source" and "Target" environments (e.g., moving from a low-traffic "Source" to a high-density "Target") using statistically accurate parameters rather than arbitrary numbers.

Key Insights and Ideas

  • The Safety-Throughput Trade-off: The AI learned a nuanced policy: when the probability of an "unexpected event" (like a car darting out from a blind spot) is high, it aggressively prioritizes radar, even if it kills data throughput.
  • Transfer is Better than "Direct Reuse": Simply copying an old AI policy to a new city (Direct Policy Reuse) performs poorly because it's too rigid. The TLwD approach uses the old data as a starting point but remains flexible enough to update its weights based on new, local observations.
  • Efficiency of Prioritization: One of the "aha" moments is that not all experiences are equal. By using prioritized replay, the AI focuses on "near-miss" scenarios or rare weather events, which accelerates the training process significantly compared to uniform sampling.

Key Results

  • Safety Gains: The proposed framework reduced the "obstacle miss detection probability" by up to 67% compared to conventional deep reinforcement learning approaches.
  • Faster Adaptation: In new environments, the TLwD algorithm reached an optimal policy in roughly 150 episodes, whereas standard DDQN required over 400 episodes to reach the same performance level.
  • Robustness: The system maintained low miss-detection rates even as traffic density increased five-fold, demonstrating that the learned policy scales well to crowded urban scenarios.

Limitations and Future Work

While the paper provides a robust framework, a few areas remain open for exploration:

  • Discrete vs. Continuous: The current model uses discrete "time steps" and a fixed set of radar/data modes. In reality, these parameters could be adjusted on a continuous scale.
  • Multi-Agent Coordination: The study focuses on a single AV's decisions. Future work could look at how a fleet of AVs might coordinate their JRC schedules so they aren't all "blinded" by data transmission at the same moment.
  • Hardware Constraints: The authors assume the switch between radar and comms is instantaneous; real-world hardware latency might introduce additional challenges not fully captured in the simulation.

Summary updated

Abstract

Autonomous Vehicles (AVs) are required to operate safely and efficiently in dynamic environments. For this, the AVs equipped with Joint Radar-Communications (JRC) functions can enhance the driving safety by utilizing both radar detection and data communication functions. However, optimizing the performance of the AV system with two different functions under uncertainty and dynamic of surrounding environments is very challenging. In this work, we first propose an intelligent optimization framework based on the Markov Decision Process (MDP) to help the AV make optimal decisions in selecting JRC operation functions under the dynamic and uncertainty of the surrounding environment. We then develop an effective learning algorithm leveraging recent advances of deep reinforcement learning techniques to find the optimal policy for the AV without requiring any prior information about surrounding environment. Furthermore, to make our proposed framework more scalable, we develop a Transfer Learning (TL) mechanism that enables the AV to leverage valuable experiences for accelerating the training process when it moves to a new environment. Extensive simulations show that the proposed transferable deep reinforcement learning framework reduces the obstacle miss detection probability by the AV up to 67% compared to other conventional deep reinforcement learning approaches. With the deep reinforcement learning and transfer learning approaches, our proposed solution can find its applications in a wide range of autonomous driving scenarios from driver assistance to full automation transportation.
BibTeX
@article{hieu2022transferable,
  author = {Nguyen Quang Hieu and Dinh Thai Hoang and Dusit Niyato and Ping Wang and Dong In Kim and Chau Yuen},
  title = {{Transferable Deep Reinforcement Learning Framework for Autonomous Vehicles With Joint Radar-Data Communications}},
  journal = {IEEE Trans. Communications},
  year = {2022},
  doi = {10.1109/tcomm.2022.3182034},
}

Loading related content...

Figure