Abstract
Autonomous Vehicles (AVs) are required to operate safely and efficiently in dynamic environments. For this, the AVs equipped with Joint Radar-Communications (JRC) functions can enhance the driving safety by utilizing both radar detection and data communication functions. However, optimizing the performance of the AV system with two different functions under uncertainty and dynamic of surrounding environments is very challenging. In this work, we first propose an intelligent optimization framework based on the Markov Decision Process (MDP) to help the AV make optimal decisions in selecting JRC operation functions under the dynamic and uncertainty of the surrounding environment. We then develop an effective learning algorithm leveraging recent advances of deep reinforcement learning techniques to find the optimal policy for the AV without requiring any prior information about surrounding environment. Furthermore, to make our proposed framework more scalable, we develop a Transfer Learning (TL) mechanism that enables the AV to leverage valuable experiences for accelerating the training process when it moves to a new environment. Extensive simulations show that the proposed transferable deep reinforcement learning framework reduces the obstacle miss detection probability by the AV up to 67% compared to other conventional deep reinforcement learning approaches.
Introduction
Modern Autonomous Vehicles (AVs) rely on Joint Radar-Communications (JRC) systems, where radar sensing and data transmission share hardware and frequency bands, such as the 71–76 GHz mm-wave band. While this integration saves space and power, it introduces a critical trade-off: time spent on data transmission is time lost for radar scanning. In dynamic environments—characterized by varying weather, traffic density, and road conditions—maintaining an optimal balance is essential for safety. Moreover, traditional AI models often struggle when an AV moves between different environments (e.g., from a rural road to an urban center), requiring costly retraining. This paper addresses these challenges by proposing a transferable deep reinforcement learning framework that optimizes JRC decisions and accelerates adaptation to new environments.
Methodology
The researchers modeled the AV’s decision-making process as a Markov Decision Process (MDP) and developed a solution using two primary techniques: Double Deep Q-Networks (DDQN) and Transfer Learning with Demonstrations (TLwD).
Double Deep Q-Network (DDQN)
To ensure stable learning in complex environments, the framework employs a DDQN architecture. Unlike standard Q-learning, which can suffer from over-optimistic value estimations, DDQN uses two separate neural networks—an Online Network and a Target Network—to decouple action selection from evaluation. The system evaluates five environmental factors: channel state, road condition, weather, vehicle speed, and nearby traffic density to choose between radar modes (long-range vs. high-resolution short-range) and data transmission rates.
Transfer Learning with Demonstrations (TLwD)
The core innovation is the TLwD mechanism, which enables the AV to transfer knowledge from a "Source" environment to a "Target" environment. Instead of starting from scratch, the AV utilizes demonstration data and Prioritized Experience Replay to focus on the most critical past experiences. Multi-step Temporal Difference (TD) learning allows the agent to look further ahead, facilitating faster convergence to an optimal policy in the new setting.
Experimental Results
The framework was evaluated using the nuScenes dataset, providing realistic traffic and environmental distributions from Boston and Singapore.
- Safety Gains: The proposed framework reduced the obstacle miss detection probability by up to 67% compared to conventional deep reinforcement learning approaches.
- Adaptation Speed: In new environments, the TLwD algorithm reached an optimal policy in approximately 150 episodes, whereas standard DDQN required over 400 episodes.
- Robustness: The learned policy remained effective even as traffic density increased five-fold, demonstrating high scalability for urban scenarios.
Conclusion
This research presents a robust framework for optimizing JRC functions in AVs. By combining DDQN with a novel transfer learning mechanism, the system achieves significant improvements in both safety and training efficiency. Future work will focus on continuous parameter optimization and multi-agent coordination to further enhance the performance of autonomous fleets in complex, shared environments.
Summary updated