DVS-RG: Differential Variable Speed Limits Control using Deep Reinforcement Learning with Graph State Representation

Jingwen Yang, Ping Wang, Fatemeh Golpayegani, Wang ShenarXiv 2024 · DOI · arXiv

TL;DR

DVS-RG is a novel deep reinforcement learning framework that optimizes freeway traffic by setting different speed limits for individual lanes. By representing the road network as a graph, the system captures complex spatial dependencies between lanes, resulting in a staggering 68.44% reduction in average waiting times and a 15.93% improvement in safety compared to existing methods.

What Problem Does This Paper Address?

Traffic bottlenecks—caused by lane drops or on-ramps—are more than just an annoyance; they lead to "stop-and-go" shockwaves, increased emissions, and higher accident risks. Variable Speed Limits (VSL) are a common tool to manage these bottlenecks by slowing upstream traffic to prevent congestion.

However, most current VSL systems apply the same speed limit across all lanes. Differential VSL (DVSL), which sets lane-specific limits, is more precise but significantly harder to control. Existing Deep Reinforcement Learning (DRL) approaches for DVSL often treat traffic data as a flat matrix, ignoring the inherent "graph" structure of a road (how lanes connect and influence their neighbors). This paper aims to bridge that gap by integrating graph-based spatial awareness into the AI's decision-making process.

Methods

The authors introduce DVS-RG (Differential Variable Speed Limits with Graph State Representation), which consists of three main components:

  1. Graph State Representation: Instead of a simple list of numbers, the road is modeled as a directed graph. Each lane is a node containing features like occupancy and average speed. Edges represent the relationships between lanes (e.g., "Lane A is upstream of Lane B" or "Lane A is next to Lane B").
  2. Message Passing & Aggregation: The model uses graph neural network techniques to allow nodes to "talk" to their neighbors. This captures how a slowdown in one lane will eventually impact the others, providing the AI with a "finer-grained" understanding of traffic flow.
  3. Actor-Critic DRL (PPO): The system uses the Proximal Policy Optimization (PPO) algorithm. The "Actor" decides the speed limits for each lane, while the "Critic" evaluates how well those decisions worked based on a reward function that balances efficiency (average speed) and safety (Time-to-Collision).

Datasets

The study utilizes SUMO (Simulation of Urban MObility), a high-fidelity microscopic traffic simulator.

  • Scenario: A classic freeway bottleneck featuring five main lanes, an on-ramp, and an off-ramp.
  • Traffic Profile: Five hours of simulated traffic with three different route choices, modeled as a Poisson process to mimic real-world variability.
  • Comparison: The model was tested against several baselines, including a "No-VSL" scenario, rule-based systems, and other state-of-the-art DRL algorithms like DDPG and TD3.

Key Insights and Ideas

  • Spatial Context is King: The "aha moment" here is that traffic isn't just a time series; it's a spatial network. By using a graph, the agent learns that a bottleneck in the merging area requires specific speed adjustments in the upstream "application area" to prevent a total standstill.
  • Differential Control Reduces Shockwaves: By allowing different speeds per lane, the system can "harmonize" flow. For example, it can slow down the lanes closest to an on-ramp to allow merging vehicles in more smoothly, while keeping other lanes moving faster.
  • Stability through PPO: While older methods like DDPG often struggle to converge (the AI "gives up" or fluctuates wildly), the PPO-based DVS-RG showed a much smoother and more stable learning curve.

Key Results

The DVS-RG model outperformed all other tested methods across the board:

  • Efficiency: Average Waiting Time (AWT) was reduced by 68.44% compared to the No-VSL baseline. Total Stopped Time (TST) dropped by 40.41%.
  • Safety: The number of potential collisions (measured via a "Near-Potential Collision" index based on Time-to-Collision) was reduced by 15.93%.
  • Throughput: The system achieved a slight increase in the number of vehicles passing through the bottleneck (+0.69%), proving that you can improve safety without sacrificing flow.

Limitations and Future Work

The authors acknowledge several areas for improvement:

  • Perfect Information Assumption: The current model assumes perfect V2X (Vehicle-to-Everything) communication with zero latency and no packet loss—a luxury not yet available in the real world.
  • Scale: The experiments were conducted on a single bottleneck. Future work needs to test how this graph-based approach handles massive, interconnected urban networks.
  • Robustness: The study does not yet account for "anomalous data" such as sensor failures or sudden emergency vehicle interventions. Evaluating how the AI reacts when its "eyes" (sensors) fail is a critical next step for deployment.

Summary updated

Abstract

Variable speed limit (VSL) control is an established yet challenging problem to improve freeway traffic mobility and alleviate bottlenecks by customizing speed limits at proper locations based on traffic conditions. Recent advances in deep reinforcement learning (DRL) have shown promising results in solving VSL control problems by interacting with sophisticated environments. However, the modeling of these methods ignores the inherent graph structure of the traffic state which can be a key factor for more efficient VSL control. Graph structure can not only capture the static spatial feature but also the dynamic temporal features of traffic. Therefore, we propose the DVS-RG: DRL-based differential variable speed limit controller with graph state representation. DVS-RG provides distinct speed limits per lane in different locations dynamically. The road network topology and traffic information(e.g., occupancy, speed) are integrated as the state space of DVS-RG so that the spatial features can be learned. The normalization reward which combines efficiency and safety is used to train the VSL controller to avoid excessive inefficiencies or low safety. The results obtained from the simulation study on SUMO show that DRL-RG achieves higher traffic efficiency (the average waiting time reduced to 68.44\%) and improves the safety measures (the number of potential collision reduced by 15.93\% ) compared to state-of-the-art DRL methods.
BibTeX
@misc{yang2024dvs,
  author = {Jingwen Yang and Ping Wang and Fatemeh Golpayegani and Wang Shen},
  title = {{DVS-RG: Differential Variable Speed Limits Control using Deep Reinforcement Learning with Graph State Representation}},
  year = {2024},
  eprint = {2405.09163},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2405.09163},
}
Figure