Flow: A Modular Learning Framework for Mixed Autonomy Traffic

Cathy Wu, Aboudy Kreidieh, Kanaad Parvate, Eugene Vinitsky, Alexandre M Bayen

TL;DR

This paper introduces Flow, an open-source, modular framework that connects deep reinforcement learning (RL) libraries with traffic micro-simulators to design and evaluate vehicle controllers for "mixed autonomy" traffic. The authors demonstrate that replacing just one human driver with an autonomous vehicle (AV)—representing a mere 4-7% adoption rate—can completely eliminate human-induced stop-and-go traffic waves, boosting average system velocity by up to 41%.

What Problem Does This Paper Address?

While the ultimate goal of the autonomous vehicle industry might be a fully automated transportation network, the transition period—where a small fraction of AVs shares the road with a majority of human drivers—presents a massive, poorly understood challenge known as mixed autonomy.

Analyzing and controlling mixed autonomy traffic is notoriously difficult. Human driving behavior introduces highly non-linear, sub-optimal dynamics (like phantom traffic jams or stop-and-go waves). Traditional traffic engineering relies on classical control theory and closed-form mathematical models (like ODEs and PDEs). To keep the math tractable, researchers are forced to make severe simplifying assumptions, usually restricting their analysis to single-lane roads or ignoring lane-changing behaviors entirely.

This paper addresses this gap by proposing a model-free approach. By leveraging deep reinforcement learning, the authors decouple the mathematical modeling of complex traffic dynamics from the design of the vehicle control laws.

Methods

The authors present Flow, a modular framework that acts as a bridge between microscopic traffic simulators (specifically SUMO) and distributed RL libraries (Ray RLlib and OpenAI Gym).

Flow allows researchers to compose traffic scenarios using reusable, "LEGO-like" components:

  • Network: Defines the physical road layout (e.g., ring roads, intersections, or real-world maps imported via OpenStreetMap).
  • Actors: The entities in the environment, which can be human-driven vehicles (modeled using standard car-following models like the Intelligent Driver Model), AVs, or infrastructure like traffic lights.
  • Observer & Control Laws: Defines what the AV can sense (e.g., local headway and velocity) and how it acts (via neural network policies like MLPs or GRUs).

Using this framework, the authors train AV controllers using Trust Region Policy Optimization (TRPO), a policy gradient method well-suited for continuous control tasks.

Fig. 1: Example network modules supported by the Flow framework. Top left: Single-lane circular track. Top middle: Multi-lane circular track. Top right: Figure-eight road network. Bottom left: Intersection network. Bottom middle: Closed loop merge network. Bottom right: Imported San Francisco network. In Flow, scenarios can be generated using OpenStreetMap (OSM) data and vehicle dynamics models from the traffic microsimulation package SUMO.

Datasets

In the context of traffic control simulation, "datasets" refer to the benchmark networks and traffic scenarios used for training and evaluation. Flow supports several standard and newly introduced benchmarks:

  1. Single-lane circular track: Based on the famous Sugiyama et al. physical experiment, which proves that human driving naturally degrades into stop-and-go waves even without bottlenecks.
  2. Multi-lane circular track: Introduces the discontinuous complexity of lane-changing.
  3. Figure-eight network: A closed network featuring a central intersection governed by right-of-way rules.
  4. San Francisco network: A real-world urban road network imported via OpenStreetMap data to demonstrate scalability.

Key Insights and Ideas

  • The Blessing of Partial Observability: Counterintuitively, restricting the AV's observation space to just its immediate surroundings (local headway and velocity) rather than providing full system-wide visibility decreased training time from 24 hours to 6 hours. Limiting the data helped the RL algorithm converge faster by removing extraneous noise, without sacrificing control performance.
  • Zero-Calibration Generalization: Classical model-based controllers (like FollowerStopper) require precise, manual tuning for specific traffic densities and fail when conditions change. In contrast, the deep RL controllers trained across a range of track lengths generalized seamlessly to entirely unseen traffic densities.
  • Emergent Platooning: In multi-AV scenarios, the neural networks implicitly learned to "platoon"—driving with smaller headways than humans safely could—which significantly increased roadway utilization and pushed system velocity past classical theoretical limits.

Key Results

  • Ring Road (Single AV, 4-7% adoption): The learned MLP and GRU controllers completely smoothed out stop-and-go waves, matching the theoretical upper bound for uniform flow. This resulted in a 36% to 41% improvement in average system velocity over the all-human baseline, outperforming classical hand-designed controllers.
  • Multi-lane Track: With 6 AVs out of 44 vehicles, the framework successfully learned continuous acceleration and discrete lane-changing behaviors simultaneously, achieving a 52.5% velocity improvement over human baselines.
  • Intersections: In the figure-eight intersection scenario, introducing a single AV improved traffic speed by 57.4%. Moving to full autonomy (100% AVs) nearly tripled the average velocity, jumping from 5 m/s to 14 m/s by eliminating intersection queues.

Limitations and Future Work

Despite its success, the authors highlight several limitations that pave the way for future research:

  • Sample Efficiency: Deep RL requires millions of simulation steps, making it computationally expensive to scale to massive, city-wide networks.
  • Safety Guarantees: Model-free RL policies lack the provable stability and safety bounds of classical control theory. Currently, Flow relies on simulator-enforced "failsafes" to prevent crashes during training.
  • Sim-to-Real Gap: Transitioning these policies from a simulator like SUMO to real-world autonomous hardware remains an open challenge, particularly when accounting for vehicle actuation delays and sensor noise.

Summary updated

Abstract

The rapid development of autonomous vehicles (AVs) holds vast potential for transportation systems through improved safety, efficiency, and access to mobility. However, the progression of these impacts, as AVs are adopted, is not well understood. Numerous technical challenges arise from the goal of analyzing the partial adoption of autonomy: partial control and observation, multi-vehicle interactions, and the sheer variety of scenarios represented by real-world networks. To shed light into near-term AV impacts, this article studies the suitability of deep reinforcement learning (RL) for overcoming these challenges in a low AV-adoption regime. A modular learning framework is presented, which leverages deep RL to address complex traffic dynamics. Modules are composed to capture common traffic phenomena (stop-and-go traffic jams, lane changing, intersections). Learned control laws are found to improve upon human driving performance, in terms of system-level velocity, by up to 57% with only 4-7% adoption of AVs. Furthermore, in single-lane traffic, a small neural network control law with only local observation is found to eliminate stop-and-go traffic - surpassing all known model-based controllers to achieve near-optimal performance - and generalize to out-of-distribution traffic densities.
BibTeX
@article{wu2022flow,
  author = {Cathy Wu and Aboudy Kreidieh and Kanaad Parvate and Eugene Vinitsky and Alexandre M Bayen},
  title = {{Flow: A Modular Learning Framework for Mixed Autonomy Traffic}},
  journal = {IEEE Trans. Robotics},
  year = {2022},
  doi = {10.1109/tro.2021.3087314},
}
Figure