Scheduling Out-of-Coverage Vehicular Communications Using Reinforcement Learning

Taylan Sahin, Ramin Khalili, Mate Boban, Adam Wolisz

TL;DR

This paper introduces VRLS, a novel reinforcement learning (RL)-based scheduler designed to maintain reliable vehicle-to-vehicle (V2V) communications in areas without cellular coverage. By proactively learning optimal resource allocation policies in simulated environments, VRLS significantly outperforms existing distributed methods, especially in challenging, high-load scenarios.

What Problem Does This Paper Address?

Vehicle-to-vehicle (V2V) communications are crucial for road safety and efficiency, enabling applications like autonomous driving and collision avoidance. While centralized scheduling by cellular base stations (BSs) offers high reliability within coverage areas, V2V communications often face disruptions in "out-of-coverage" (OOC) zones, such as tunnels or areas with sparse infrastructure. In these OOC areas, vehicles typically resort to less efficient distributed scheduling methods (like 3GPP's Mode 4 or IEEE 802.11p). These distributed methods suffer from issues like the "hidden node problem," leading to reduced reliability and performance. The challenge is to provide consistent, high-quality V2V communication even when cellular coverage is lost, without requiring constant re-training or complex on-the-fly adaptation in real-world, dynamic environments.

Methods

The core of this paper is the Vehicular Reinforcement Learning Scheduler (VRLS). VRLS is a centralized RL agent that operates before vehicles enter an OOC zone. It learns a policy to proactively assign time-frequency resources for V2V transmissions within these delimited OOC areas (DOCAs).

Key methodological aspects include:

  • Reinforcement Learning (RL) Framework: VRLS uses a deep neural network (DNN) to represent its policy. It learns by interacting with a simulated vehicular environment, receiving rewards based on the quality of its resource assignments. The training utilizes an actor-critic RL algorithm.
  • Unified State Representation: VRLS is designed to handle diverse OOC environments by using a normalized state representation that accounts for resource pool configuration, vehicle density, and spatial information (distance to the DOCA entrance).
  • Proactive Scheduling: The scheduler is positioned within the network infrastructure (e.g., at the edge of the OOC zone) and assigns resources to vehicles before they enter the OOC area, based on information available from the delimiting BSs.
  • Simulation-Based Training: To overcome the cost and complexity of real-world training, VRLS is trained in simplified and simulated environments. The paper demonstrates that a policy trained in simulation can be effectively deployed in more complex, realistic environments without further training.
  • Data Augmentation: To make the learned policy invariant to the order of resources in the state representation, data augmentation techniques like random shuffling of resources are employed.
  • Handling Multiple Resource Pools: A novel approach is presented where a single VRLS policy can be trained to handle multiple different resource pool configurations (varying numbers of time-frequency resources) simultaneously, by training with a "master" pool configuration and using parallel workers for different configurations.

Datasets

The paper does not explicitly mention using pre-existing public datasets. Instead, it relies on:

  • Simulated Environments: The training and evaluation are conducted using custom-built simulation environments.
  • Training Environment (E0): A simplified environment with basic mobility and wireless channel characteristics to enable efficient training.
  • Evaluation Environments (E1, E2): More realistic environments built using the ns-3 network simulator and SUMO traffic simulator, incorporating realistic mobility, path loss, shadowing, and V2V traffic models as defined by 3GPP. These environments vary in lane configurations, DOCA lengths, and vehicle densities (loaded and highly-loaded scenarios).

Key Insights and Ideas

  • RL for Proactive OOC Scheduling is Viable: The paper convincingly demonstrates that RL can learn effective resource allocation policies for V2V communications in OOC zones, a task that is challenging for traditional methods.
  • Simulation-to-Reality Transfer is Effective: A policy trained in a simplified simulation environment can generalize well to more complex, realistic scenarios, significantly reducing the need for costly real-world training.
  • Centralized Proactive Scheduling Outperforms Distributed Reactive Methods: VRLS's ability to assign resources before entering an OOC zone provides a significant advantage over reactive distributed methods like Mode 4, especially in high-density and high-load conditions.
  • Unified Policy for Diverse Resource Pools: The ability to train a single RL agent to manage various resource pool configurations is a significant practical advantage, reducing deployment complexity.
  • Addressing the Hidden Node Problem: VRLS's learned policy implicitly addresses the hidden node problem by intelligently assigning resources, particularly by differentiating between vehicles in the same and opposite directions.

Key Results

  • Reliability Improvement: VRLS reduces packet loss by half compared to Mode 4 in highly loaded conditions and achieves near-maximum reliability in low-load scenarios.
  • Awareness Probability: VRLS enables higher awareness probability (PA) for V2V applications, allowing for greater communication ranges compared to Mode 4. For example, in E2-HL with LDOCA=500m, VRLS achieves a 120m awareness range, while Mode 4 achieves only 80m.
  • Fairness and Latency: VRLS maintains similar user fairness and latency as Mode 4, while offering superior reliability.
  • Aperiodic Traffic Performance: VRLS's learned policy for periodic traffic is also applicable to aperiodic (event-triggered) traffic, showing good performance.
  • High Power and Larger Pool Performance: VRLS demonstrates high reliability even with increased transmit power and larger resource pools, showing adaptability.
  • Multi-Configuration Handling: VRLS achieves PRR close to theoretical bounds when trained on multiple resource pool configurations simultaneously.

Limitations and Future Work

  • HD Constraint Learning: While VRLS is effective, learning the nuances of the half-duplex (HD) constraint in large state spaces can still be challenging, leading to some HD errors.
  • Unforeseen Conditions: The paper acknowledges that VRLS might not be trained for all possible scenarios (e.g., unexpected accidents causing severe congestion).
  • Future Work: The authors suggest a hybrid approach combining VRLS with distributed sensing-based scheduling for OOC areas. They also mention congestion control mechanisms and resource over-provisioning as potential solutions for unforeseeable conditions. Further work is also suggested to evaluate the network efficiency and resource demand of VRLS under varying coverage conditions.

Summary updated

Abstract

Performance of vehicle-to-vehicle (V2V) communications depends highly on the employed scheduling approach. While centralized network schedulers offer high V2V communication reliability, their operation is conventionally restricted to areas with full cellular network coverage. In contrast, in out-of-cellular-coverage areas, comparatively inefficient distributed radio resource management is used. To exploit the benefits of the centralized approach for enhancing the reliability of V2V communications on roads lacking cellular coverage, we propose VRLS (Vehicular Reinforcement Learning Scheduler), a centralized scheduler that proactively assigns resources for out-of-coverage V2V communications <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">before</i> vehicles leave the cellular network coverage. By training in simulated vehicular environments, VRLS can learn a scheduling policy that is robust and adaptable to environmental changes, thus eliminating the need for targeted (re-)training in complex real-life environments. We evaluate the performance of VRLS under varying mobility, network load, wireless channel, and resource configurations. VRLS outperforms the state-of-the-art distributed scheduling algorithm in zones without cellular network coverage by reducing the packet error rate by half in highly loaded conditions and achieving near-maximum reliability in low-load scenarios.
BibTeX
@misc{sahin2022scheduling,
  author = {Taylan Sahin and Ramin Khalili and Mate Boban and Adam Wolisz},
  title = {{Scheduling Out-of-Coverage Vehicular Communications Using Reinforcement Learning}},
  year = {2022},
  eprint = {2207.06537},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2207.06537},
}
Figure