Abstract
Traditional lateral control in autonomous vehicles often relies on Model Predictive Control (MPC), which provides precision but suffers from high computational costs and sensitivity to model inaccuracies. This paper presents a hybrid approach using Reinforcement Learning from Demonstration (RLfD) to train a Deep Reinforcement Learning (DRL) agent. By utilizing an MPC-PID controller as an online demonstrator within the CARLA simulator, the proposed system overcomes the "cold start" problem of DRL. The resulting controller achieves a 78.13% reduction in computational time compared to traditional MPC while maintaining high precision and demonstrating superior robustness to sensor noise and generalization to unseen environments like the Monza racing track.
1. Introduction
Lateral control—the ability of an autonomous vehicle (AV) to accurately follow a planned trajectory—is a fundamental challenge in robotics. Current methodologies generally fall into two categories: model-based control and learning-based control. Model-based approaches, such as Model Predictive Control (MPC), utilize mathematical vehicle models to optimize steering inputs. While precise, these methods are computationally intensive and require meticulous tuning.
Conversely, learning-based approaches like Deep Reinforcement Learning (DRL) offer adaptability and can handle complex, non-linear scenarios without an explicit model. However, DRL agents often struggle with the "cold start" problem, where initial random exploration leads to inefficient training and frequent collisions. This research proposes a hybrid framework that leverages the strengths of both paradigms by using a traditional MPC-PID setup to guide a DRL agent through its early training phases.
2. Methodology
The researchers implemented a Deep Deterministic Policy Gradient (DDPG) agent within the CARLA simulation environment. The core innovation lies in the integration of an "online demonstrator" to facilitate Reinforcement Learning from Demonstration (RLfD).
2.1 Online Guidance and the Extra Action Mechanism
Unlike standard imitation learning, which relies on static, pre-recorded datasets, this system employs an MPC-PID controller that runs in parallel with the DRL agent during training. An "Extra Action" mechanism allows the system to execute the MPC-PID’s steering command instead of the agent's command based on a specific probability. This ensures the agent remains within "successful" states, maximizing the efficiency of the training process and preventing the agent from spending excessive time in failure states (e.g., crashing).
2.2 Reward Function and Cyclical Learning Rates
The reward function was redesigned to incorporate an imitation penalty. In addition to standard lane-keeping rewards, the agent is penalized if its steering output deviates significantly from the expert MPC-PID command ($c_{diff}$). This weight is adaptive: early in training, the agent is incentivized to mimic the expert; as performance improves, the focus shifts toward optimizing track-specific performance. To further enhance training stability and avoid local optima, the authors utilized a cyclical learning rate strategy, which oscillates the step size to find the best parameters more efficiently.
3. Experimental Setup
The system was developed and tested using the CARLA simulator, an industry-standard for autonomous driving research. Training was conducted on custom maps featuring diverse geometries, including straight sections and sharp turns. For evaluation, the researchers used two distinct environments:
- Circular Track: A standardized circuit with a 50-meter radius to measure low-speed precision.
- Digital Twin: A high-fidelity model of the Autodromo Nazionale Monza racing circuit to test the agent’s ability to generalize to unseen, high-speed environments.
4. Results and Analysis
The hybrid DRL agent demonstrated significant improvements over traditional control methods across several key metrics.
4.1 Computational Efficiency
One of the most striking results was the reduction in computational overhead. The DRL agent completed the test track in 145.28 seconds, compared to 664.23 seconds for the MPC-PID controller—a 78.13% improvement. This efficiency stems from the fact that once trained, the DRL agent functions as a feed-forward neural network, bypassing the complex real-time optimization required by MPC.
4.2 Precision and Robustness
On the circular test circuit, the agent achieved an Average Lateral Error (ALE) of only 0.00283 meters. Furthermore, the agent exhibited remarkable robustness to sensor noise. When random "jitter" was introduced to the input waypoints to simulate sensor errors, the DRL agent maintained a stable trajectory, whereas traditional controllers often struggle with such inconsistent inputs.
4.3 Generalization
The agent successfully navigated the Monza racing track without prior exposure to the environment. This indicates that the RLfD process allowed the agent to learn fundamental driving principles and predictive steering behaviors—such as smoothing out corners by observing future waypoints—rather than simply memorizing training paths.
5. Conclusion and Future Work
This study demonstrates that Reinforcement Learning from Demonstration can bridge the gap between traditional control theory and modern machine learning. By using an MPC-PID controller as a temporary "teacher," the authors produced a DRL agent that is faster, more robust, and highly precise.
Future research will focus on integrating longitudinal control (speed) with lateral control into a single unified DRL framework. Additionally, the authors aim to explore "Sim-to-Real" transfer to validate these findings on physical vehicle hardware under varying real-world weather and lighting conditions.
Summary updated