Hi-CoLA: High Confidence Lower Bound Approximation Based Reinforcement Learning for Flex-Route Transit Operation Control

Qi Kang, Haris N. Koutsopoulos, Hongliang Guo, Jinhua ZhaoarXiv 2026 · arXiv

TL;DR

Hi-CoLA (High Confidence Lower Bound Approximation) is a new reinforcement learning framework designed to make real-time transit control both efficient and statistically "safe." By using a differentiable neural network to approximate performance lower bounds, it allows transit agencies to optimize bus routes with a mathematical guarantee that the new policy won't perform worse than the current one.

What Problem Does This Paper Address?

Flex-Route Transit (FRT) is a hybrid transportation model where buses follow a fixed base route but can deviate to pick up on-demand passengers at "flexible stops." While this offers a great balance between efficiency and service, managing these deviations in real-time is a nightmare of high-dimensional variables like traffic, passenger no-shows, and varying demand.

Reinforcement Learning (RL) is a natural fit for this kind of dynamic control, but it faces a major "trust gap." In public infrastructure, a policy that performs well on average but occasionally causes a catastrophic service collapse is unacceptable. Most RL methods lack statistical guarantees; deploying them is, as the authors put it, "as uncertain as tossing a coin." Hi-CoLA aims to bridge this gap by providing a "deploy-or-not" certificate based on user-defined confidence levels.

Methods

The core innovation of Hi-CoLA is how it handles High-Confidence Off-Policy Evaluation (HCOPE). Standard HCOPE uses importance sampling and concentration inequalities to calculate a performance lower bound, ensuring that with probability $1-\delta$, the actual performance is above that bound. However, HCOPE is computationally expensive and non-differentiable, making it difficult to use directly for policy optimization.

Hi-CoLA solves this through a two-phase process:

  1. Surrogate Training: The researchers pre-train a "Hi-CoLA Network" (a multi-layer perceptron). They perturb the current behavioral policy to create a set of candidate policies, calculate their HCOPE lower bounds, and train the network to map policy parameters directly to these bounds.
  2. Differentiable Optimization: Once trained, this network acts as a fast, differentiable proxy. The system can then use standard policy gradient methods to maximize the performance lower bound.

To keep the math honest, the framework operates within a trust region, using a KL-divergence filter to ensure the new policy doesn't drift too far from the data-supported behavioral policy, which would make the importance sampling estimates unreliable.

The Hi-CoLA framework workflow, showing the pre-training and improvement phases.

Datasets

The authors evaluated Hi-CoLA using:

  • Synthetic FRT Environments: Simplified scenarios with 3 to 5 fixed stops and multiple flexible stops to test monotonic improvement and baseline comparisons.
  • Chicago Case Study: A high-fidelity simulation of an actual bus route in the Chicago Medical District using the SUMO (Simulation of Urban MObility) simulator. This involved modeling real-world constraints like 15-minute headways and stochastic passenger arrivals.

Key Insights and Ideas

  • Pessimism as a Feature: By optimizing for the lower bound rather than the expected return, the agent becomes naturally conservative. It only chooses to deviate the bus if it is highly confident that doing so will improve ridership without risking service reliability.
  • Amortizing Computation: The use of a neural network surrogate "amortizes" the heavy lifting of statistical evaluation. This allows for rapid, iterative policy updates that would be impossible if one had to run full HCOPE calculations at every gradient step.
  • Robustness to "Bad" Starts: A surprising finding was that Hi-CoLA is highly robust to the quality of the initial policy. Even if the starting "behavioral" policy is poor or highly random, the framework can extract useful signal to find a safe path to improvement.

Key Results

Hi-CoLA was benchmarked against state-of-the-art offline RL (like CQL and IQL) and safe RL (Daedalus).

  • Tail-Risk Performance: In terms of Conditional Value at Risk ($CVaR_{5\%}$), which measures the worst 5% of outcomes, Hi-CoLA achieved an 11.30% improvement over the baseline, significantly outperforming CQL (8.55%) and IQL (7.57%).
  • Reliability: As shown in the performance distribution plots, Hi-CoLA not only achieved the highest median ridership but also showed a much tighter spread, indicating more predictable and reliable service.
  • Monotonic Improvement: The training process demonstrated a clear, steady increase in both the 90% confidence lower bound and actual policy performance, proving the "guaranteed improvement" claim.
Fig. 5: Performance Distribution Comparison

Limitations and Future Work

While Hi-CoLA excels at single-agent control, the authors note that real-world transit often involves multiple vehicles that need to coordinate. Future work will focus on extending the framework to multi-agent settings. Additionally, while the SUMO simulation is high-fidelity, the ultimate test will be integrating this into large-scale, live operational data pipelines where data noise and sensor errors are even more prevalent.

Summary updated

Abstract

This paper studies safe policy improvement for real-time operation control of flex-route transit (FRT) systems, where robustness is critical and any performance decrease is undesirable. While reinforcement learning (RL) has shown strong empirical success, its adoption remains limited in safety and cost critical applications, such as intelligent transportation, since learned policies lack statistically certified performance guarantees before deployment. In this paper, we propose the High Confidence Lower Bound Approximation (Hi-CoLA) RL framework, which combines importance sampling and concentration inequalities with a differentiable surrogate network to efficiently estimate the high-confidence performance lower bound and optimize it via policy gradient for robust performance improvement. On the FRT operation control problem, Hi-CoLA outperforms state-of-the-art offline and safe RL methods in both average and tail-risk performance. A case study on an actual route in Chicago, implemented in the SUMO simulator, serves as a proof of concept. Hi-CoLA enhances real-time FRT control with guaranteed performance and is broadly applicable to other safety critical decision-making scenarios.
BibTeX
@misc{kang2026hi,
  author = {Qi Kang and Haris N. Koutsopoulos and Hongliang Guo and Jinhua Zhao},
  title = {{Hi-CoLA: High Confidence Lower Bound Approximation Based Reinforcement Learning for Flex-Route Transit Operation Control}},
  year = {2026},
  eprint = {2611.01071},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.01071},
}
Figure