Deep Reinforcement Learning-Based Resource Allocation for Cellular Vehicular Network Mode 3 with Underlay Approach

Jinjuan Fu, Xizhong Qin, Yan Huang, Li Tang, Yan Liu2022 · DOI

TL;DR

This paper tackles the challenge of efficiently allocating radio resources in cellular vehicular networks (V2X) using an underlay approach. They propose a novel deep reinforcement learning (DRL) framework that allows V2V (vehicle-to-vehicle) communication to share spectrum with V2I (vehicle-to-infrastructure) links, optimizing V2I throughput while meeting V2V latency demands. A key innovation is using partial CSI (channel state information) to reduce signaling overhead, achieving performance comparable to full CSI.

What Problem Does This Paper Address?

Modern transportation systems are increasingly relying on vehicle-to-vehicle (V2V) and vehicle-to-infrastructure (V2I) communications for safety and efficiency. Cellular V2X (C-V2X) is a promising technology for this, with different modes of operation. Mode 3, specifically, involves the base station (BS) managing resource allocation. When V2V links share the same spectrum as V2I links (an "underlay" approach), interference becomes a significant issue. The challenge is to allocate these shared resources intelligently to maximize the data rate for V2I users while ensuring that V2V communications, which are highly sensitive to delay, meet their strict latency requirements. Traditional methods struggle with the complexity and dynamic nature of vehicular environments, and obtaining complete channel state information (CSI) for all users at the BS is impractical due to high signaling overhead.

Methods

The authors propose a Deep Reinforcement Learning (DRL) framework for resource allocation in Mode 3 underlay C-V2X. Specifically, they employ a Double Deep Q-Network (DQN) algorithm, which is known for its stability and ability to handle complex state spaces.

Here's a breakdown of their approach:

  • Agent: The Base Station (BS) acts as the learning agent.
  • Environment: The vehicular network, including V2I and V2V links, is the environment.
  • State Space: Instead of requiring the BS to have complete CSI of all V2I and V2V links, they design a partial CSI observation space. Each V2V user feeds back interference channel power gain and latency information to the BS. This significantly reduces the signaling overhead. The BS's state is an aggregation of these partial CSI observations from all V2V pairs.
  • Action Space: The BS's action is to assign subcarriers to V2V pairs.
  • Reward Function: The reward function is designed to balance two objectives:
  • Maximizing the sum capacity of V2I links.
  • Ensuring V2V links meet their latency requirements (modeled as successful delivery of a payload within a time budget). A separate reward component is defined for V2V latency satisfaction.
  • Double DQN with Experience Replay: They utilize an experience replay buffer to store past interactions (state, action, reward, next state) and a mini-batch sampling for training. The use of a "target network" separate from the "training network" in Double DQN helps to mitigate overestimation bias and improve training stability.

Datasets

Not explicitly discussed in the paper. The authors mention using a single-cell scenario and following the simulation setup from a previous work ([22]) for their experiments.

Key Insights and Ideas

  • Partial CSI is Sufficient: The most significant insight is that using only partial CSI (interference and latency information from V2V links) at the BS is sufficient to achieve performance comparable to using complete CSI. This is crucial for reducing signaling overhead in dynamic vehicular environments.
  • DRL for Dynamic Resource Management: DRL proves to be an effective tool for making intelligent, real-time resource allocation decisions in a complex and dynamic vehicular network, adapting to changing channel conditions and user demands.
  • Balancing Conflicting Objectives: The proposed reward function effectively balances the competing goals of maximizing V2I throughput and meeting V2V latency constraints, demonstrating the power of RL in multi-objective optimization.
  • Mode 3 Underlay Viability: The work demonstrates the practical feasibility of the Mode 3 underlay approach for C-V2X, showing it can be efficient when managed by an intelligent DRL-based system.

Key Results

  • The proposed DRL-based scheme with partial CSI meets the low latency requirements of V2V links.
  • It increases the capacity of V2I links compared to other methods.
  • The performance of the proposed scheme using partial CSI is comparable to that using complete CSI.
  • The proposed scheme's V2V success probability is close to brute-force methods and significantly better than random allocation.
  • The V2I link throughput achieved by the proposed scheme is higher than the random scheme and competitive with other DRL-based approaches.
  • The algorithm shows better stability in highly dynamic environments (e.g., increasing vehicle speeds) compared to random allocation.

Limitations and Future Work

  • Simulation-Based: The results are based on simulations, and real-world deployment might present additional challenges.
  • Single-Cell Scenario: The current model focuses on a single-cell scenario. Future work could explore multi-cell interference and coordination.
  • Specific DRL Algorithm: While Double DQN is effective, exploring other advanced DRL algorithms might yield further improvements.
  • Channel Models: The paper uses specific channel models; testing with more diverse and realistic channel models could be beneficial.
  • Payload Size Variation: While tested, a more in-depth analysis of how the algorithm adapts to a wider range of payload sizes and their impact on latency could be explored.

Summary updated

BibTeX
@misc{fu2022deep,
  author = {Jinjuan Fu and Xizhong Qin and Yan Huang and Li Tang and Yan Liu},
  title = {{Deep Reinforcement Learning-Based Resource Allocation for Cellular Vehicular Network Mode 3 with Underlay Approach}},
  year = {2022},
  doi = {10.3390/s22051874},
}
Figure