Optimizing Transportation Dynamics at a City-Scale Using a Reinforcement Learning Framework

Luckyson Khaidem, Massimiliano Luca, Fan Yang, Ankit Anand, Bruno Lepri, Wen Dong

TL;DR: Researchers have developed a model-based reinforcement learning framework that optimizes city-wide traffic flow by approximating multi-agent simulations with a Partially Observable Discrete Event Decision Process (PODEDP), significantly reducing travel times in real-world urban datasets.

Introduction: The Challenge of Urban Scale

Urban traffic optimization is plagued by a "scale gap." While current tools can effectively manage a single intersection or a small neighborhood, they often fail when applied to the dynamics of an entire city due to exploding computational complexity. Furthermore, testing new traffic policies in real-world settings is both dangerous and expensive. This research introduces a model-based reinforcement learning (RL) framework capable of bridging the gap between high-fidelity transportation simulators and scalable optimization.

Methodology: Integrating MATSim and PODEDP

The core of this framework is the integration of the Multi-Agent Transport Simulation (MATSim) with a novel RL algorithm based on a Partially Observable Discrete Event Decision Process (PODEDP).

Realistic Simulation with MATSim

Unlike simplified simulators, MATSim agents possess "plans"—they follow daily schedules like going to work or shopping. This creates realistic traffic patterns that policy-makers can trust. The authors transformed MATSim into a reinforcement learning "gym" environment, allowing modern ML agents to interact with complex urban dynamics.

The PODEDP Model

To avoid the data-inefficiency of model-free RL, the researchers adopted a model-based approach:

  • Discrete Event: Traffic is modeled as a series of events (e.g., a vehicle moving from one road link to another) rather than a continuous fluid flow.
  • Partially Observable: The model realistically assumes that only a fraction of vehicles (e.g., "probe" vehicles with GPS) are visible to the controller, mirroring real-world data constraints.
  • Variational Inference: To handle the "curse of dimensionality" in city-wide state spaces, the authors used mean-field and Bethe entropy approximations to make the complex mathematics tractable.

Experimental Validation

The framework was validated across four diverse scenarios to ensure robustness:

  1. SynthTown: A synthetic benchmark for initial validation.
  2. Berlin, Germany: A network featuring 2,400 links and 17,000 synthesized vehicles.
  3. Santiago de Chile: A larger network with 6,000 links and 65,000 vehicles.
  4. Dakar, Senegal: A real-world scenario with 8,000 links and 12,000 vehicles, derived from Call Detail Records (CDR).

Key Insights and Performance

The PODEDP approach consistently outperformed state-of-the-art benchmarks, including Policy Gradient, Actor-Critic, and Guided Policy Search.

  • Efficiency and Stability: Model-based learning proved significantly more data-efficient and stable than model-free techniques, which often suffered from high variance and slow convergence.
  • The Power of Probes: The study found that observing just 10% of vehicles as "probes" provides sufficient information to optimize the behavior of the entire population.
  • Metric Improvements: In the Berlin scenario, the framework reduced the average Time on Road (TOR) to 80 minutes, compared to 679 minutes for Actor-Critic methods. It also maximized the number of Vehicles at Work (VAW) during business hours, indicating superior punctuality.
  • High Fidelity: Quantile-quantile (Q-Q) plots demonstrated that the PODEDP mathematical approximation almost perfectly matched the high-fidelity MATSim dynamics, ensuring the model captures the "essence" of urban movement.

Conclusion and Future Directions

This framework offers a convincing tool for urban planners by providing realistic, scalable, and data-efficient traffic policies. While highly promising, future work remains to bridge the final "sim-to-real" gap through live deployment. Additionally, integrating heterogeneous data sources—such as public transit schedules and pedestrian flows—and exploring decentralized multi-agent controllers for mega-cities represent the next frontiers for this research.

Summary updated

Abstract

Urban planners, authorities, and numerous additional players have to deal with challenges related to the rapid urbanization process and its effect on human mobility and transport dynamics. Hence, optimize transportation systems represents a unique occasion for municipalities. Indeed, the quality of transport is linked to economic growth, and by decreasing traffic congestion, the life quality of the inhabitants is drastically enhanced. Most state-of-the-art solutions optimize traffic in specific and small zones of cities (e.g., single intersections) and cannot be used to gather insights for an entire city. Moreover, evaluating such optimized policies in a realistic way that is convincing for policy-makers can be extremely expensive. In our work, we propose a reinforcement learning frameworks to overtake these two limitations. In particular, we use human mobility data to optimize the transport dynamics of three real-world cities (i.e., Berlin, Santiago de Chile, Dakar) and a synthesized one (i.e., SynthTown). To this end, we transform the transportation dynamics' simulator MATSim into a realistic reinforcement learning environment able to optimize and evaluate transportation policies using agents that perform realistic daily activities and trips. In this way, we can assess transportation policies in a manner that is convincing for policy-makers. Finally, we develop a model-based reinforcement learning algorithm that approximates MATSim dynamics with a Partially Observable Discrete Event Decision Process (PODEDP) and, with respect to other state-of-art policy optimization techniques, can scale on big transportation data and find optimal policies also on a city-scale.
BibTeX
@article{khaidem2020optimizing,
  author = {Luckyson Khaidem and Massimiliano Luca and Fan Yang and Ankit Anand and Bruno Lepri and Wen Dong},
  title = {{Optimizing Transportation Dynamics at a City-Scale Using a Reinforcement Learning Framework}},
  journal = {IEEE Access},
  year = {2020},
  doi = {10.1109/access.2020.3024979},
}
Figure