Abstract
This paper presents a safe learning-based eco-driving framework tailored for mixed traffic flows, which aims to optimize energy efficiency while guaranteeing safety during real-system operations. Even though reinforcement learning (RL) is capable of optimizing energy efficiency in intricate environments, it is challenged by safety requirements during the training process. The lack of safety guarantees is the other concern when deploying a trained policy in real-world application. Compared with RL, model predicted control (MPC) can handle constrained dynamics systems, ensuring safe driving. However, the major challenges lie in complicated eco-driving tasks and the presence of disturbances, which respectively challenge the MPC design and the satisfaction of constraints. To address these limitations, the proposed framework incorporates the tube-based enhanced MPC (RMPC) to ensure the safe execution of the RL policy under disturbances, thereby improving the control robustness. RL not only optimizes the energy efficiency of the connected and automated vehicle in mixed traffic but also handles more uncertain scenarios, in which the energy consumption of the human-driven vehicle and its diverse and stochastic driving behaviors are considered in the optimization framework. Simulation results demonstrate that the proposed algorithm, compared with RMPC technique, shows an average improvement of 10.88% in holistic energy efficiency, while compared with RL algorithm, it effectively prevents inter-vehicle collisions.
Introduction
Eco-driving aims to optimize vehicle trajectories to minimize energy consumption. While Reinforcement Learning (RL) excels at finding optimal policies in complex, stochastic environments, it faces two major hurdles for real-world automotive deployment: safety during the trial-and-error training phase and a lack of formal guarantees when faced with disturbances like sensor noise or unexpected human maneuvers. Conversely, Model Predictive Control (MPC) provides mathematical safety guarantees but struggles with highly nonlinear energy models and the stochastic nature of human driving behavior.
This paper introduces a hybrid control framework that combines the optimization power of RL with the rigorous safety guarantees of Robust Model Predictive Control (RMPC). Designed for mixed traffic—where autonomous vehicles (CAVs) coexist with human drivers—the system achieves a 10.88% improvement in energy efficiency over standard controllers while effectively eliminating collision risks.
Methodology
The proposed architecture functions as an RMPC-certified RL framework, acting as a "safety filter" for autonomous decision-making.
Core Components:
- The RL Agent (TD3): A Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm acts as the high-level controller, suggesting acceleration commands to optimize a multi-objective reward function including energy economy, traffic efficiency, and safety.
- The Safety Filter (Tube-based RMPC): Instead of applying the RL action directly, it is passed through a Tube-based Robust MPC. This component "tightens" the physical constraints of the vehicle to account for bounded disturbances. If the RL's suggested action would violate safety constraints, the RMPC modifies it to the closest safe alternative.
- Human Behavior Estimation: Using a Least Squares method, the system online-identifies the driving style (specifically the time headway) of the following human-driven vehicle (HDV). This allows the RL agent to optimize the "holistic" energy of the small platoon, rather than just the CAV's individual consumption.
Experimental Setup
The study utilizes the Next Generation Simulation (NGSIM) dataset for both calibration and evaluation. The authors used 1,737 car-following cases to calibrate the Intelligent Driver Model (IDM), ensuring simulated human drivers reflect real-world behavioral diversity. Performance was then verified across 1,626 specific cases to test the framework against a wide range of human driving preferences.
Results and Discussion
The framework was tested in simulations against pure RMPC and pure RL controllers, yielding three primary insights:
- Holistic Efficiency: A key finding is that a CAV should drive in a way that induces efficient behavior in the following HDV. By smoothing its own speed, the CAV prevents the "stop-and-go" waves that waste energy in trailing vehicles.
- Safety and Robustness: While pure RL suffered from multiple collisions during learning, the RMPC-certified version maintained zero collisions throughout training and deployment. The system remained robust even in the presence of stochastic disturbances and measurement noise.
- Performance Gains: The RMPC-certified RL showed an average 10.88% improvement in holistic energy efficiency compared to standard RMPC techniques.
Conclusion
By decoupling optimization from safety, this framework allows RL agents to explore aggressive energy-saving strategies without the risk of collisions. Future work will focus on scaling this approach to larger multi-lane platoons and intersections, as well as exploring the impact of communication delays and non-electric vehicle energy models.
Summary updated