Multi-Agent Deep Deterministic Policy Gradient-Based Coordinated Control for Urban Expressway Entrance–Arterial Interfaces

Shunchao Wang, Zhigang Wu, Wenwu YuSystems 2026 · DOI

TL;DR

This paper tackles the complex problem of traffic congestion at urban expressway-arterial interfaces by proposing a novel multi-agent reinforcement learning (MARL) approach. By coordinating variable speed limits (VSL), ramp metering (RM), and intersection signals (ISC) using MADDPG, the system significantly reduces travel times, delays, and shockwave propagation, leading to smoother and more efficient traffic flow.

What Problem Does This Paper Address?

Urban expressway-arterial interfaces are notorious bottlenecks where different traffic flow regimes (high-speed expressways and slower arterials) meet. This transition zone is further complicated by the interplay of three key control mechanisms: variable speed limits on the expressway, ramp metering to control vehicle entry onto the expressway, and traffic signals at the arterial intersection. Existing control strategies for these individual components often operate in isolation, leading to fragmented responses and an inability to adapt to the complex, heterogeneous traffic demands. This lack of coordination can result in cascading congestion, shockwave propagation, and overall reduced network efficiency. The paper aims to bridge this gap by developing a unified, cooperative control framework that jointly optimizes these three actuators to achieve system-level traffic flow improvements.

Methods

The core of this research lies in a Multi-Agent Deep Deterministic Policy Gradient (MADDPG) framework. Here's a breakdown of the key methodological components:

  • Multi-Agent Reinforcement Learning (MARL): The system is framed as a cooperative multi-agent problem, with three distinct agents: one for VSL, one for RM, and one for ISC. Each agent learns its own policy but contributes to a shared global objective.
  • Centralized Training with Decentralized Execution (CTDE): The MADDPG algorithm allows for centralized training where a global critic can access all agents' observations and actions to stabilize learning. However, during real-time execution, each agent acts independently based on its local observations, enabling fast response times.
  • Asynchronous Control Cycle Mechanism: Recognizing that VSL, RM, and ISC operate on different temporal scales (e.g., VSL updates every 30s, RM every 5s), a novel asynchronous mechanism is introduced. This synchronizes decisions onto a common 5s grid while allowing agents to hold their actions when not due for an update, preserving feasibility and avoiding excessive switching.
  • Conflict-Aware Reward Design: A shaped reward function is employed to guide the agents. It combines a global reward (net boundary flow) with agent-specific rewards and penalty terms. These penalties address critical aspects like density overshoot, speed variance, spillback risk, and actuation costs, ensuring a balanced optimization that prevents subsystem-specific gains at the expense of overall system performance.
  • Compact State and Action Spaces: The agents operate with carefully designed state and action spaces that capture essential traffic dynamics without being overly complex, facilitating efficient learning and real-time execution.

Datasets

The study utilizes a calibrated urban network model in SUMO (Simulation of Urban Mobility). This model is based on a real-world site in Changsha, Hunan, and is calibrated using the Intelligent Driver Model (IDM) and the LC2013 model. The training and evaluation scenarios are configured with multiple demand levels, spanning moderate to heavily congested periods, to assess the robustness of the proposed framework. No new datasets were introduced; rather, a realistic simulation environment was created.

Key Insights and Ideas

  • The Power of Coordination: The most significant insight is that true system-level efficiency at these interfaces comes not from optimizing individual controllers aggressively, but from their intelligent, coordinated interplay. The MADDPG framework effectively orchestrates these agents to act as a "dynamic buffer."
  • Asynchronous Control is Feasible and Beneficial: The paper demonstrates that managing heterogeneous control cycles asynchronously, rather than forcing synchronized updates, is a practical and effective approach. This preserves feasibility and avoids unnecessary overhead.
  • Shaped Rewards for Holistic Optimization: The carefully designed reward function, incorporating both flow-based contributions and penalty terms for stability and feasibility, is crucial. It prevents agents from optimizing locally at the expense of the global system and ensures robust performance.
  • Real-time Deployment Potential: The focus on lightweight actor networks for online inference and the asynchronous scheduling mechanism highlight the framework's suitability for real-time deployment on existing traffic control hardware.

Key Results

The proposed MADDPG-based coordinated control strategy demonstrated significant improvements over classical baselines:

  • Mainline Merge: Average travel time reduced to 13.56 s, which is 62.4% of the VSL-only approach.
  • Ramp: Occupancy lowered to 6.4%, representing 40.6% of the ALINEA (ramp-only) control.
  • Signalized Approach: Average delay decreased to 85.71 s, which is 62.7% of actuated control.
  • Congestion Onset and Propagation: The framework effectively delays congestion onset and significantly reduces shockwave propagation on the ramp and expressway.
  • Throughput Improvement: Overall throughput at the interface is improved.
  • Robustness: The system shows graceful degradation with sensor noise and is tolerant to moderate inaccuracies.

Limitations and Future Work

  • Focus on Single Intersection: The current study focuses on a single expressway-arterial interface with one adjacent signalized intersection. The authors acknowledge that extending this to more complex multi-intersection corridors, with their unique spacing and control dynamics, is a crucial next step.
  • Further Ablation Studies: More systematic analyses of the shaped reward components and their weights are suggested to better quantify the impact of different penalties.
  • Benchmarking against Unified Controllers: While compared against mature single-strategy baselines, future work could benchmark against contemporary RL/MPC controllers that employ unified state/action designs and timing assumptions.
  • Statistical Robustness: More comprehensive statistical evaluations with varied random seeds and demand realizations are needed to further quantify performance variability.

Summary updated

Abstract

Coordinated control of ramp metering, variable speed limits, and intersection signals is critical for mitigating congestion and enhancing efficiency at urban expressway–arterial interfaces. Existing strategies often operate in isolation, leading to fragmented responses and limited adaptability under heterogeneous traffic demands. This study develops a multi-agent reinforcement learning framework based on MADDPG to achieve cooperative decision-making across heterogeneous controllers. An asynchronous control cycle mechanism is designed to accommodate different temporal requirements of ramp meters, speed limits, and signal controllers, ensuring practical feasibility in real-time operations. A conflict-aware reward design further embeds density regulation, speed harmonization, and spillback prevention to stabilize flow dynamics. Simulation experiments on a calibrated urban network demonstrate that the proposed framework delays congestion onset, reduces shockwave propagation, and improves throughput compared with classical benchmarks. In particular, at the mainline merge, average travel time is reduced to 13.56 s (62.4% of VSL-only); at the ramp, occupancy is lowered to 6.4% (40.6% of ALINEA); and at the signalized approach, average delay decreases to 85.71 s (62.7% of actuated control). These results highlight the scalability and deployment potential of the proposed cooperative control approach for system-level traffic management in mixed traffic environments.
BibTeX
@inproceedings{wang2026multi,
  author = {Shunchao Wang and Zhigang Wu and Wenwu Yu},
  title = {{Multi-Agent Deep Deterministic Policy Gradient-Based Coordinated Control for Urban Expressway Entrance–Arterial Interfaces}},
  booktitle = {Systems},
  year = {2026},
  doi = {10.3390/systems14030231},
}
Figure