Robust Dynamic Bus Control: A Distributional Multi-agent Reinforcement Learning Approach

Jiawei Wang, Lijun Sun

TL;DR

This paper introduces IQNC-M, a novel distributional multi-agent reinforcement learning (MARL) framework designed to make bus holding control robust against real-world transit uncertainties. By integrating implicit quantile networks with a meta-learning approach for adaptive distortion weights, IQNC-M effectively handles traffic perturbations, demand surges, and service disruptions, leading to more reliable and efficient bus operations.

What Problem Does This Paper Address?

Bus systems are crucial for urban mobility, but their operation is inherently unstable due to unpredictable traffic conditions and passenger demand. A major consequence of this instability is "bus bunching," where buses arrive in groups, severely degrading service reliability and efficiency. While multi-agent reinforcement learning (MARL) has shown promise in controlling bus holding (i.e., making buses wait at stops to space them out), existing MARL approaches often fail to account for the robustness required for real-world deployment. They struggle with various real-world events like traffic jams, service interruptions, and sudden demand spikes, which can lead to cascading failures. This paper aims to bridge this gap by developing a MARL framework that explicitly addresses robustness against these uncertainties.

Methods

The core of the proposed solution is the IQNC-M framework, which builds upon distributional reinforcement learning (DRL) and MARL.

  • Distributional Reinforcement Learning (DRL): Instead of learning the expected value of states and actions, DRL learns the entire distribution of these values. This allows the agent to better capture and reason about the inherent uncertainties in the transit system. The paper utilizes an Implicit Quantile Network (IQN) to model this state-action value distribution.
  • Multi-Agent Reinforcement Learning (MARL): Each bus is treated as an independent agent that makes decisions (holding duration) based on its local observations. The challenge here is the "credit assignment problem" – determining how each agent's actions contribute to the overall system performance, especially when actions are asynchronous.
  • Meta-Learning for Adaptive Distortion Weights: To address the non-stationarity and credit assignment issues in MARL, the authors introduce a meta-learning module. This module learns to dynamically adjust "distortion weights." These weights are applied to the state-action value distribution, allowing the agent to express its confidence in its own contribution to the system's performance. Higher confidence (risk-seeking) leads to different distortions than lower confidence (risk-averse). The meta-learner adapts these weights based on global information (asynchronous events from other buses), making the policy evaluation more robust and flexible.
  • Actor-Critic Architecture: The framework uses an actor-critic setup. The critic learns the distorted state-action value distribution, and the actor uses this evaluation to learn a robust holding control policy.

Datasets

The study uses real-world bus service data from four selected bus routes in an Asian city. This data includes smart card records for passenger demand and travel times. The authors developed and calibrated a bus simulator to reproduce the patterns of real-world operation based on this data.

Key Insights and Ideas

  • Robustness is Paramount: The paper strongly emphasizes that for any AI-driven control system to be deployed in the real world, robustness against unexpected events is not just desirable but essential.
  • Distributional RL for Uncertainty: Learning the distribution of outcomes, rather than just the average, is a powerful way to equip agents with a better understanding of uncertainty, crucial for making safer and more reliable decisions in dynamic environments.
  • Meta-Learning for Adaptive Confidence: The novel meta-learning approach to adapt distortion weights is a key innovation. It allows agents to dynamically adjust their "risk preference" or confidence in their actions based on the broader system context, directly tackling the credit assignment problem in a more nuanced way than fixed risk measures.
  • Global Information for Local Decisions: By incorporating global information (events from other buses) through the meta-learner, the framework enables agents to make more informed local decisions, leading to better coordinated system-level performance.

Key Results

The proposed IQNC-M framework demonstrated superior performance across various metrics and scenarios:

  • Traffic State Perturbations: IQNC-M achieved the best regularity with moderate increases in travel time, outperforming traditional methods and other MARL baselines, including those using distributional RL without meta-learning.
  • Transit Demand Perturbations: Similar to traffic perturbations, IQNC-M excelled in stabilizing the system (smaller average waiting time and occupancy dispersion) under varying demand levels.
  • Anomaly Scenarios (Traffic Interruption & Demand Surge): Visualizations of bus trajectories showed that IQNC-M was significantly better at recovering from and maintaining system performance during anomalies compared to other methods.
  • Transferability: The model showed good performance even on routes where it was not explicitly trained, indicating good generalization capabilities.
  • Interpretability: Analysis of the meta-learned distortion weights showed that the model learned to place higher confidence on the ego bus's contribution when there were fewer other control events, providing an interpretable mechanism for its robustness.

Limitations and Future Work

  • Limitations:
    • While the framework handles various perturbations, the paper acknowledges that the "risk-seeking" and "risk-averse" settings (IQNC-CF and IQNC-UCF) without meta-learning were less effective than IQNC-M, suggesting that adaptive distortion is crucial.
  • The paper notes that strategies focusing solely on minimizing waiting time might not be optimal in all severe traffic situations.
  • Future Work:
    • Adversarial Reinforcement Learning: Exploring adversarial RL to train agents against an adversary that intentionally creates perturbations could further enhance robustness.
  • Multi-Line Operation: Extending the framework to coordinate control across multiple bus routes simultaneously, which share infrastructure, is a significant next step.
  • Generalizability: The authors believe the core idea of adaptive distortion weights for MARL could be applied to other multi-agent systems beyond transit operations.

Summary updated

Abstract

Bus system is a critical component of sustainable urban transportation. However, the operation of a bus fleet is unstable in nature, and bus bunching has become a common phenomenon that undermines the efficiency and reliability of bus systems. Recently research has demonstrated the promising application of multi-agent reinforcement learning (MARL) to achieve efficient vehicle holding control to avoid bus bunching. However, existing studies essentially overlook the robustness issue resulting from various events, perturbations and anomalies in a transit system, which is of utmost importance when transferring the models for real-world deployment/application. In this study, we integrate implicit quantile network and meta-learning to develop a distributional MARL framework -- IQNC-M -- to learn continuous control. The proposed IQNC-M framework achieves efficient and reliable control decisions through better handling various uncertainties/events in real-time transit operations. Specifically, we introduce an interpretable meta-learning module to incorporate global information into the distributional MARL framework, which is an effective solution to circumvent the credit assignment issue in the transit system. In addition, we design a specific learning procedure to train each agent within the framework to pursue a robust control policy. We develop simulation environments based on real-world bus services and passenger demand data and evaluate the proposed framework against both traditional holding control models and state-of-the-art MARL models. Our results show that the proposed IQNC-M framework can effectively handle the various extreme events, such as traffic state perturbations, service interruptions, and demand surges, thus improving both efficiency and reliability of the system.
BibTeX
@article{wang2021robust,
  author = {Jiawei Wang and Lijun Sun},
  title = {{Robust Dynamic Bus Control: A Distributional Multi-agent Reinforcement Learning Approach}},
  journal = {IEEE Trans. ITS},
  year = {2021},
  doi = {10.1109/tits.2022.3229527},
}
Figure