Integration of Decentralized Graph-Based Multi-Agent Reinforcement Learning with Digital Twin for Traffic Signal Optimization

Vijayalakshmi K. Kumarasamy, Abhilasha Saroj, Yu Liang, Dalei Wu, Michael P. Hunter, Angshuman Guin, Mina SartipiSymmetry 2024 · DOI

TL;DR

This paper introduces a novel approach to optimize traffic signal timing by integrating a decentralized graph-based multi-agent reinforcement learning (DGMARL) system with a Digital Twin. This combination allows for more adaptive and efficient traffic management, leading to significant reductions in fuel consumption and congestion, particularly during peak hours.

What Problem Does This Paper Address?

Optimizing traffic signal timing in complex urban environments is a significant challenge. Traditional methods often struggle to adapt to dynamic traffic conditions, leading to congestion, increased travel times, and higher fuel consumption. Existing machine learning approaches, while promising, can face scalability and adaptability issues in large traffic networks. This paper aims to address these limitations by developing a more intelligent and responsive traffic signal control system that can learn and adapt in real-time.

Methods

The core of this research lies in the integration of two key technologies:

  • Decentralized Graph-Based Multi-Agent Reinforcement Learning (DGMARL): This approach treats each intersection as an independent agent that learns to optimize its traffic signal timing. Agents communicate with their neighbors to share local observations and coordinate their actions, forming a decentralized network. The "graph-based" aspect means the relationships and connections between intersections are explicitly modeled, allowing for more sophisticated learning.
  • Digital Twin: A Digital Twin is a virtual replica of the physical traffic network. It receives real-time data from sensors and traffic infrastructure, allowing the DGMARL agents to interact with a simulated environment that accurately reflects real-world conditions. This provides a safe and efficient platform for training and testing the reinforcement learning agents without disrupting actual traffic.

The integration allows DGMARL agents to learn from the Digital Twin's simulations, making informed decisions about signal phasing to minimize a custom metric called "Eco_PI," which quantifies fuel consumption and stop delays. The system also incorporates constraints like minimum green time and pedestrian recall to ensure safety and realism.

Datasets

The study utilizes a real-world dataset collected from the Martin Luther King (MLK) Smart Corridor in Chattanooga, Tennessee, USA. This dataset includes:

  • Roadway network geometry
  • Traffic signal timing plans
  • Data from cameras and zone-detecting devices
  • Signal Phasing and Timing (SPaT) data
  • Vehicle flow, speed, and presence time in detector zones

This data was used to build a PTV-Vissim simulation model of the 11-intersection corridor, which served as the Digital Twin.

Key Insights and Ideas

  • Digital Twins as a Training Ground: The Digital Twin acts as a crucial intermediary, providing a realistic and dynamic environment for training RL agents. This bypasses the need for extensive and potentially disruptive real-world testing during the initial learning phases.
  • Decentralized Learning for Scalability: By distributing the decision-making process across multiple agents (intersections), the system can scale more effectively to larger and more complex traffic networks compared to centralized approaches.
  • Eco_PI as a Holistic Performance Metric: The development and use of the Eco_PI metric, which combines stop delays and penalized stops, provides a more comprehensive evaluation of traffic efficiency and environmental impact than traditional metrics.
  • Dynamic Phasing is Key: The ability of the DGMARL system to dynamically adjust signal phases based on real-time traffic demand, rather than relying on fixed timings, is fundamental to its improved performance.

Key Results

The proposed DGMARL system demonstrated significant improvements over the baseline actuated signal timing plan:

  • Overall 24-hour scenario: A 55.38% reduction in the Eco_PI index was observed across the 10 intersections.
  • PM-peak hour scenario: A 38.94% average reduction in Eco_PI was achieved over ten test runs.
  • Specific improvements: Reductions in Eco_PI ranged from 3.17% to 62.14% across different intersections.
  • Stop Delays: Average stop delays decreased by 42.78% during PM-peak hours.
  • Vehicles Arriving on Green: Some intersections saw an increase in vehicles arriving on green, indicating smoother flow. For example, Douglas intersection saw a 17.48% increase.

Limitations and Future Work

  • Contextual Differences: The authors acknowledge that their findings might differ from previous studies due to variations in traffic patterns, infrastructure, and environmental factors across different contexts.
  • Further Investigation Needed: The occasional instances of higher Eco_PI at the Douglas intersection compared to the actuated scenario suggest a need for further investigation into the specific factors influencing these outcomes.
  • Future Research Directions:
    • Extensive testing in real-road environments.
  • Deploying the algorithm in larger traffic networks.
  • Incorporating additional functionalities like adaptive learning mechanisms and varying optimization frequencies.
  • Investigating the algorithm's robustness under diverse traffic scenarios and its scalability.

Summary updated

BibTeX
@article{kumarasamy2024integration,
  author = {Vijayalakshmi K. Kumarasamy and Abhilasha Saroj and Yu Liang and Dalei Wu and Michael P. Hunter and Angshuman Guin and Mina Sartipi},
  title = {{Integration of Decentralized Graph-Based Multi-Agent Reinforcement Learning with Digital Twin for Traffic Signal Optimization}},
  journal = {Symmetry},
  year = {2024},
  doi = {10.3390/sym16040448},
}
Figure