SocialLight: Distributed Cooperation Learning towards Network-Wide Traffic Signal Control

Harsh Goel, Yifeng Zhang, Mehul Damani, Guillaume Sartoretti

Abstract

Urban traffic congestion poses significant economic and environmental challenges. While Adaptive Traffic Signal Control (ATSC) offers a potential solution, existing Reinforcement Learning (RL) methods struggle with the trade-off between scalability and cooperation. This paper introduces SocialLight, a scalable multi-agent reinforcement learning (MARL) framework that enables traffic signals to cooperate by estimating their individual "marginal contribution" to their local neighborhood. By combining a locally-centralized critic with counterfactual reasoning and stabilized advantage estimates, SocialLight achieves state-of-the-art performance on large-scale real-world road networks, including a 196-intersection grid in New York City.

Introduction

Adaptive Traffic Signal Control (ATSC) aims to mitigate urban congestion by adjusting signal timings in real-time based on traffic conditions. However, designing effective control strategies for large-scale networks is difficult. Centralized RL methods suffer from the "curse of dimensionality" as the action space grows exponentially with the number of intersections. Conversely, independent RL methods, where each intersection acts as an autonomous agent, often lead to suboptimal "greedy" behaviors that ignore downstream impacts.

Figure 1: Intersection layout and signal phases used in the ATSC environment.

Previous cooperative MARL approaches, such as COMA, utilize a global critic to address credit assignment but fail to scale to city-wide networks. SocialLight bridges this gap by distributing the credit assignment problem spatially, focusing on local neighborhoods to ensure both cooperation and scalability.

Methodology

SocialLight is built upon the Asynchronous Actor-Critic (A3C) framework, enhanced by three primary innovations designed for scalable cooperation:

1. Locally-Centralized Critic

Instead of a global critic observing the entire network or a purely local critic, SocialLight employs a critic conditioned on an agent’s immediate neighbors. This approach captures the most significant traffic dependencies while maintaining computational efficiency.

2. Counterfactual Reasoning

Drawing inspiration from the COMA algorithm, each agent calculates its "marginal contribution." This involves assessing how much the neighborhood reward changed specifically due to the agent's action compared to a baseline of all other possible actions, given the neighbors' actions.

3. Stabilized Advantage Calculation

To address the instability of standard counterfactual baselines in highly distributed environments, SocialLight integrates Temporal Difference (TD) bootstrapping and Generalized Advantage Estimation (GAE). This reduces variance in gradient updates, preventing neighbor-induced "noise" from destabilizing the learning process.

Figure 2: (a) The multi-agent architecture of SocialLight showing local neighborhoods and (b) the detailed structure of the Actor and Critic networks.

Experimental Evaluation

The framework was evaluated using two traffic simulators: SUMO for synthetic grids and CityFlow for real-world datasets.

  • Synthetic Manhattan (SUMO): A $5 \times 5$ grid (25 intersections) used to benchmark queue lengths and trip times.
  • Real-World Datasets (CityFlow):
    • Jinan: 12 intersections.
  • Hangzhou: 16 intersections.
  • New York City: A massive $28 \times 7$ grid (196 intersections), representing a significant scale for MARL research.

Key Results and Insights

SocialLight consistently outperformed state-of-the-art baselines, including AttentionLight, CoLight, and MPLight.

  • Performance Gains: In the 196-intersection NYC grid, SocialLight achieved a 24.53% improvement in average travel time over the best-performing baseline. In synthetic SUMO tests, it reduced average intersection delay by over 60% (from 25.43s to 10.07s).
  • Spatial Credit Assignment: The results demonstrate that global cooperation can emerge from overlapping local neighborhoods. Physical traffic flow allows cooperative signals to propagate across the network without requiring global vision.
  • Training Stability: Ablation studies confirmed that the proposed stabilized advantage calculations are critical; without them, agents failed to converge effectively in distributed settings.

Conclusion and Future Work

SocialLight proves that local centralization is a viable path to city-scale traffic management. While the current model assumes a fixed spatial structure, future research could explore dynamic neighborhood learning and the integration of heterogeneous intersection types. Additionally, investigating the dynamics between cooperative signals and autonomous vehicles presents a promising direction for future urban mobility systems.

Summary updated

BibTeX
@inproceedings{goel2023sociallight,
  author = {Harsh Goel and Yifeng Zhang and Mehul Damani and Guillaume Sartoretti},
  title = {{SocialLight: Distributed Cooperation Learning towards Network-Wide Traffic Signal Control}},
  booktitle = {AAMAS},
  year = {2023},
}
Figure