Safe Reinforcement Learning-Based Eco-Driving Control for Mixed Traffic Flows With Disturbances

Ke Lu, Dongjun Li, Qun Wang, Kaidi Yang, Lin Zhao, Ziyou Song

TL;DR

This paper introduces a hybrid control framework that combines the optimization power of Reinforcement Learning (RL) with the rigorous safety guarantees of Robust Model Predictive Control (RMPC). Designed for mixed traffic—where autonomous vehicles (CAVs) coexist with human drivers—the system achieves a 10.88% improvement in energy efficiency over standard controllers while effectively eliminating the collision risks typically associated with RL training.

What Problem Does This Paper Address?

Eco-driving aims to optimize vehicle trajectories to minimize energy consumption. While Reinforcement Learning (RL) excels at finding optimal policies in complex, stochastic environments, it has two major "deal-breakers" for real-world automotive deployment:

  1. Safety during training: RL agents learn by trial and error, which in a driving context means they might cause collisions before they learn to avoid them.
  2. Lack of guarantees: Even a trained RL policy can behave unpredictably when faced with disturbances like sensor noise or unexpected human maneuvers.

Conversely, Model Predictive Control (MPC) provides mathematical safety guarantees but struggles with the highly nonlinear energy models and the "black-box" nature of human driving behavior. The authors tackle the challenge of creating a controller that is both high-performing (efficient) and robustly safe in mixed traffic flows.

Methods

The authors propose an RMPC-certified RL framework. The architecture functions like a "safety filter":

  • The RL Agent (TD3): A Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm acts as the high-level "brain," suggesting acceleration commands to optimize a multi-objective reward function (energy economy of both the CAV and the following human vehicle, traffic efficiency, and safety).
  • The Safety Filter (Tube-based RMPC): Instead of applying the RL action directly, it is passed through a Tube-based Robust MPC. This component "tightens" the physical constraints of the vehicle to account for bounded disturbances (like measurement noise). If the RL's suggested action would violate safety constraints, the RMPC modifies it to the closest safe alternative.
  • Human Behavior Estimation: Since the CAV's behavior affects the vehicle behind it, the authors use a Least Squares method to online-identify the driving style (specifically the time headway) of the following human-driven vehicle (HDV). This allows the RL agent to optimize the "holistic" energy of the small platoon, not just itself.

Datasets

The study utilizes the Next Generation Simulation (NGSIM) dataset, a widely recognized benchmark in transportation research.

  • Calibration: The authors used 1,737 car-following cases from NGSIM to calibrate the Intelligent Driver Model (IDM), ensuring the simulated human drivers in their tests reflect real-world behavioral diversity.
  • Evaluation: 1,626 specific cases from the dataset were used to verify the eco-driving performance across a wide range of human driving preferences.

Key Insights and Ideas

  • The "Holistic" Efficiency: A key insight is that a CAV shouldn't just drive efficiently for itself; it should drive in a way that induces efficient behavior in the human-driven vehicle following it. By smoothing out its own speed, the CAV prevents the "stop-and-go" waves that waste energy in the vehicles behind.
  • Decoupling Optimization from Safety: By using RMPC as a certificate, the RL agent can explore aggressive energy-saving strategies during training without the risk of actual collisions. This "fail-safe" allows for faster and more effective learning.
  • Handling the "Human Element": The paper recognizes that human drivers aren't monolithic. By identifying the specific "driving preference" of the follower in real-time, the CAV can tailor its strategy to that specific driver's habits.

Key Results

The framework was tested in simulations against pure RMPC and pure RL controllers:

  • Energy Efficiency: The proposed RMPC-certified RL showed an average 10.88% improvement in holistic energy efficiency compared to the standard RMPC technique.
  • Safety: While the pure RL algorithm suffered from multiple collisions during its learning phase (specifically at episodes 231, 147, and 2048), the RMPC-certified version maintained zero collisions throughout training and deployment.
  • Robustness: The system successfully maintained safety and efficiency even in the presence of stochastic disturbances and measurement noise, where traditional non-robust controllers might fail.

Limitations and Future Work

While promising, the paper notes some areas for further exploration:

  • Scenario Complexity: The current study focuses on a three-vehicle platoon (Preceding Vehicle, CAV, Following Vehicle). Future work could scale this to larger, more complex multi-lane platoons or intersections.
  • Communication Assumptions: The model assumes the CAV can accurately sense the states of vehicles around it. Exploring the impact of communication delays or "hidden" vehicles (not equipped with sensors) would be a logical next step.
  • Generalization: The energy model is specific to electric vehicles; adapting the reward functions for internal combustion or hybrid vehicles would broaden the application.

Summary updated

Abstract

This paper presents a safe learning-based eco-driving framework tailored for mixed traffic flows, which aims to optimize energy efficiency while guaranteeing system constraints during real-system operations. Even though reinforcement learning (RL) is capable of optimizing energy efficiency in intricate environments, it is challenged by safety requirements during both the training and deployment stages. The lack of safety guarantees impedes the application of RL to real-world problems. Compared with RL, model predicted control (MPC) can handle constrained dynamics systems, ensuring safe driving. However, the major challenges lie in complicated eco-driving tasks and the presence of disturbances, which pose difficulties for MPC design and constraint satisfaction. To address these limitations, the proposed framework incorporates the tube-based enhanced MPC (RMPC) to ensure the safe execution of the RL policy under disturbances, thereby improving the control robustness. RL not only optimizes the energy efficiency of the connected and automated vehicle in mixed traffic but also handles more uncertain scenarios, in which the energy consumption of the human-driven vehicle and its diverse and stochastic driving behaviors are considered in the optimization framework. Simulation results demonstrate that the proposed algorithm achieves an average improvement of 10.88% in holistic energy efficiency compared to the RMPC technique, while effectively preventing inter-vehicle collisions when compared to the RL algorithm.
BibTeX
@article{lu2025safe,
  author = {Ke Lu and Dongjun Li and Qun Wang and Kaidi Yang and Lin Zhao and Ziyou Song},
  title = {{Safe Reinforcement Learning-Based Eco-Driving Control for Mixed Traffic Flows With Disturbances}},
  journal = {IEEE Trans. Intell. Transp. Syst.},
  year = {2025},
  url = {https://openreview.net/forum?id=PjH0JaCTIt},
}

Loading related content...

Figure