Robust Reinforcement Learning Strategies with Evolving Curriculum for Efficient Bus Operations in Smart Cities

Yuhan Tang, Ao Qu, Xuan Jiang, Baichuan Mo, Shangqing Cao, Joseph Rodriguez, Haris N. Koutsopoulos, Cathy Wu, Jinhua ZhaoSmart 2024 · DOI

TL;DR

This paper introduces a novel reinforcement learning (RL) framework for optimizing bus operations in smart cities. By combining three distinct actions (holding, skipping, and turning around) with an evolving curriculum and domain randomization, the approach significantly reduces passenger waiting times and improves schedule reliability, outperforming traditional methods.

What Problem Does This Paper Address?

Efficient and reliable public transit is crucial for urban life and the development of smart cities. A persistent challenge in bus operations is "bus bunching," where buses arrive too close together, leading to uneven service and increased passenger wait times. While reinforcement learning (RL) has been explored to address this, existing methods often oversimplify the problem by focusing on single strategies like "holding" buses at stops. These approaches struggle with the inherent complexities and uncertainties of real-world traffic, passenger demand, and temporal dynamics, leading to suboptimal solutions that don't effectively minimize overall passenger time in the system. This paper aims to tackle these limitations by developing a more robust and comprehensive RL strategy.

Methods

The core of this research is a novel RL framework that integrates several key components:

  • Multi-Action Strategy: Instead of just "holding," the RL agent can choose from three high-level actions: "holding" (waiting at a stop), "skipping" a stop, and "turning around" to serve the opposite direction. This provides greater flexibility in managing bus flow.
  • Long Short-Term Memory (LSTM): To account for the temporal dependencies and non-Markovian nature of bus operations (e.g., how a previous bus's action affects the current one), an LSTM network is incorporated. This allows the agent to learn from sequential data and make more informed decisions.
  • Curriculum Learning: Learning all three actions simultaneously is complex. The authors employ curriculum learning, where the agent is trained incrementally. It starts with simpler actions and gradually progresses to more complex ones, allowing for more efficient learning and better generalization.
  • Domain Randomization (DR): To make the model robust to real-world uncertainties like unpredictable traffic and passenger demand, Domain Randomization is used. This involves training the agent across a variety of simulated environments with randomized parameters, ensuring it can adapt to unseen conditions.
  • Proximal Policy Optimization (PPO): The RL algorithm used is PPO, a stable and efficient actor-critic method known for its ability to handle complex environments.

Datasets

The paper mentions using both synthetic and real-world data for evaluation. Specifically, departure data was gathered from the Chicago Transit Authority (CTA), and passenger arrival rates were fitted from CTA data.

Key Insights and Ideas

  • Beyond Holding: The most significant insight is that a combination of strategies (holding, skipping, turning around) is far more effective than relying solely on holding. This multi-faceted approach allows for dynamic adjustments that traditional methods cannot achieve.
  • Curriculum is Key: Directly learning all three actions at once is challenging. The success of the curriculum learning approach, where actions are introduced progressively, highlights its importance for tackling complex action spaces in RL.
  • Robustness Through Randomization: Domain Randomization is crucial for bridging the "sim-to-real" gap. By exposing the agent to a wide range of randomized scenarios during training, the model becomes significantly more resilient to real-world variations.
  • LSTM for Temporal Dynamics: The integration of LSTM is vital for capturing the sequential nature of bus operations, enabling the agent to learn from past states and actions to make better future decisions.

Key Results

The proposed RL framework demonstrated significant improvements:

  • Reduced Passenger Waiting Time: Total passenger waiting time was reduced by over 15%.
  • Improved Schedule Reliability: Bus schedules became more evenly distributed, as visualized in Figure 6.
  • Enhanced Efficiency: The combined strategies yielded more effective results than solely relying on the holding strategy, with a reduction in total system time by 6% and average passenger waiting time by 16%.
  • Ablation Study Success: The ablation study showed that incorporating LSTM, DR, and Curriculum Learning progressively improved performance, with the full model (Scenario 5) achieving the best results.

Limitations and Future Work

  • Implementation Complexity: The authors acknowledge that implementing this model in reality would require significant investment in communication systems between buses, control centers, and passengers, as well as real-time tracking infrastructure.
  • Passenger Cooperation: They suggest exploring incentive mechanisms to encourage passenger cooperation and potentially restricting certain actions to ensure passenger convenience.
  • Future Curricula: The paper proposes exploring improved curricula using algorithmic strategies specifically tailored for bus operations.
  • Sim-to-Real Transfer: A key future goal is to assess the effectiveness of domain-randomized training for sim-to-real transfer by field-testing the algorithm in real-world conditions with a transit agency. This would involve implementing tracking systems and collecting passenger feedback.

Summary updated

Abstract

Public transit systems are critical to the quality of urban life, and enhancing their efficiency is essential for building cost-effective and sustainable smart cities. Historically, researchers sought reinforcement learning (RL) applications to mitigate bus bunching issues with holding strategies. Nonetheless, these attempts often led to oversimplifications and misalignment with the goal of reducing the total time passengers spent in the system, resulting in less robust or non-optimal solutions. In this study, we introduce a novel setting where each bus, supervised by an RL agent, can appropriately form aggregated policies from three strategies (holding, skipping station, and turning around to serve the opposite direction). It’s difficult to learn them all together, due to learning complexity, we employ domain knowledge and develop a gradually expanding action space curriculum, enabling agents to learn these strategies incrementally. We incorporate Long Short-Term Memory (LSTM) in our model considering the temporal interrelation among these actions. To address the inherent uncertainties of real-world traffic systems, we impose Domain Randomization (DR) on variables such as passenger demand and bus schedules. We conduct extensive numerical experiments with the integration of synthetic and real-world data to evaluate our model. Our methodology proves effective, enhancing bus schedule reliability and reducing total passenger waiting time by over 15%, thereby improving bus operation efficiency and smoothering operations of buses that align with sustainable goals. This work highlights the potential of robust RL combined with curriculum learning for optimizing public transport in smart cities, offering a scalable solution for real-world multi-agent systems.
BibTeX
@inproceedings{tang2024robust,
  author = {Yuhan Tang and Ao Qu and Xuan Jiang and Baichuan Mo and Shangqing Cao and Joseph Rodriguez and Haris N. Koutsopoulos and Cathy Wu and Jinhua Zhao},
  title = {{Robust Reinforcement Learning Strategies with Evolving Curriculum for Efficient Bus Operations in Smart Cities}},
  booktitle = {Smart},
  year = {2024},
  doi = {10.3390/smartcities7060141},
}
Figure