Reducing Bus Bunching with Asynchronous Multi-Agent Reinforcement Learning

Jiawei Wang, Lijun Sun

Introduction

Bus bunching is a pervasive issue in urban transit systems where buses on the same route lose their scheduled spacing, eventually arriving at stops in clusters. This phenomenon is inherently unstable: a slight delay increases the number of passengers at the next stop, further slowing the bus, while the following bus speeds up due to fewer pickups. To combat this, transit agencies use "bus holding" strategies, intentionally delaying buses at stops to restore even headways.

While Multi-Agent Reinforcement Learning (MARL) offers a promising path for optimizing these holding times, real-world bus systems are asynchronous. Unlike standard MARL environments where agents act in synchronized turns, a bus only makes a decision upon arriving at a stop. This temporal misalignment makes it difficult for traditional algorithms to assign credit—determining how a specific action by one bus at one time affects the system's overall performance later.

The Challenge of Asynchronous Control

In a typical bus route, agents (buses) operate independently and trigger control events at different times. Standard MARL critics struggle to evaluate these staggered actions because the state of the environment is constantly shifting between decision points.

Asynchronous control view: Bus $i$ determines holding time upon arrival at a stop, while other buses are at different stages of their routes.

As shown in the figure above, the control logic must account for the fact that feedback is received after a delay, and the impact of a holding action ripples through both upstream and downstream agents. To address this, researchers developed the Credit Assignment framework for Asynchronous Control (CAAC).

The CAAC Framework

The core innovation of CAAC is its ability to handle asynchronous interactions through a specialized architecture. It moves away from fixed time-steps and instead treats the system as a dynamic Event Graph (EG), where nodes represent arrival events and edges represent spatiotemporal dependencies between buses.

Dual-Critic Architecture

To effectively assign credit in this asynchronous environment, CAAC employs a dual-critic structure:

  • Ego Critic: This component evaluates the bus's own local policy and immediate state, focusing on the direct consequences of its holding actions.
  • Event Critic: Utilizing Graph Attention Networks (GAT), the Event Critic analyzes the Event Graph to calculate the "marginal contribution" of other buses. It identifies how the actions of nearby agents (within a specific time window) influence the current bus's performance.
The CAAC Dual-Critic Architecture separates the evaluation into Ego and Event contributions to estimate actual environmental feedback.

By separating "what I did" from "how others affected me," the model achieves more stable training and a clearer understanding of multi-agent coordination. Furthermore, because the GAT-based critic learns general relationships rather than specific schedules, the framework supports inductive learning, allowing a model trained on one route to be deployed on another without retraining (zero-shot transferability).

Experimental Evaluation

The framework was validated using real-world smart card data from four major trunk routes (R1–R4) in a large city. A custom simulator, calibrated to match real-world boarding patterns and travel uncertainties, served as the testing ground. CAAC was compared against several baselines, including "No Control," traditional headway-based methods, and standard MARL algorithms like MADDPG.

Key Results

  • Service Reliability: CAAC achieved the lowest Average Waiting Time (AWT) and Average Occupancy Dispersion (AOD) across all tested routes, indicating more reliable service and more evenly distributed passenger loads.
  • Robustness and Transferability: A model trained solely on Route R1 outperformed all baselines when tested on Routes R2, R3, and R4. For instance, on Route R2, CAAC significantly mitigated the increase in waiting time compared to the "No Control" scenario, demonstrating its ability to generalize across different route geometries and bus counts.

Conclusion

CAAC represents a significant step forward in applying MARL to real-world, asynchronous infrastructure. By leveraging graph-based credit assignment, it effectively manages the complex, staggered interactions inherent in transit systems. Future work may involve integrating traffic signal priority and exploring more complex passenger demand patterns to further enhance urban mobility.

Summary updated

Abstract

The bus system is a critical component of sustainable urban transportation. However, due to the significant uncertainties in passenger demand and traffic conditions, bus operation is unstable in nature and bus bunching has become a common phenomenon that undermines the reliability and efficiency of bus services. Despite recent advances in multi-agent reinforcement learning (MARL) on traffic control, little research has focused on bus fleet control due to the tricky asynchronous characteristic -- control actions only happen when a bus arrives at a bus stop and thus agents do not act simultaneously. In this study, we formulate route-level bus fleet control as an asynchronous multi-agent reinforcement learning (ASMR) problem and extend the classical actor-critic architecture to handle the asynchronous issue. Specifically, we design a novel critic network to effectively approximate the marginal contribution for other agents, in which graph attention neural network is used to conduct inductive learning for policy evaluation. The critic structure also helps the ego agent optimize its policy more efficiently. We evaluate the proposed framework on real-world bus services and actual passenger demand derived from smart card data. Our results show that the proposed model outperforms both traditional headway-based control methods and existing MARL methods.
BibTeX
@inproceedings{wang2021reducing,
  author = {Jiawei Wang and Lijun Sun},
  title = {{Reducing Bus Bunching with Asynchronous Multi-Agent Reinforcement Learning}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.24963/ijcai.2021/60},
}
Figure