Autonomous Vehicle Lateral Control Using Deep Reinforcement Learning with MPC-PID Demonstration

Chengdong Wu, Sven Kirchner, Nils Purschke, Alois Knoll

Abstract

Traditional lateral control in autonomous vehicles often relies on Model Predictive Control (MPC), which provides precision but suffers from high computational costs and sensitivity to model inaccuracies. This paper presents a hybrid approach using Reinforcement Learning from Demonstration (RLfD) to train a Deep Reinforcement Learning (DRL) agent. By utilizing an MPC-PID controller as an online demonstrator within the CARLA simulator, the proposed system overcomes the "cold start" problem of DRL. The resulting controller achieves a 78.13% reduction in computational time compared to traditional MPC while maintaining high precision and demonstrating superior robustness to sensor noise and generalization to unseen environments like the Monza racing track.

1. Introduction

Lateral control—the ability of an autonomous vehicle (AV) to accurately follow a planned trajectory—is a fundamental challenge in robotics. Current methodologies generally fall into two categories: model-based control and learning-based control. Model-based approaches, such as Model Predictive Control (MPC), utilize mathematical vehicle models to optimize steering inputs. While precise, these methods are computationally intensive and require meticulous tuning.

Conversely, learning-based approaches like Deep Reinforcement Learning (DRL) offer adaptability and can handle complex, non-linear scenarios without an explicit model. However, DRL agents often struggle with the "cold start" problem, where initial random exploration leads to inefficient training and frequent collisions. This research proposes a hybrid framework that leverages the strengths of both paradigms by using a traditional MPC-PID setup to guide a DRL agent through its early training phases.

2. Methodology

The researchers implemented a Deep Deterministic Policy Gradient (DDPG) agent within the CARLA simulation environment. The core innovation lies in the integration of an "online demonstrator" to facilitate Reinforcement Learning from Demonstration (RLfD).

2.1 Online Guidance and the Extra Action Mechanism

Unlike standard imitation learning, which relies on static, pre-recorded datasets, this system employs an MPC-PID controller that runs in parallel with the DRL agent during training. An "Extra Action" mechanism allows the system to execute the MPC-PID’s steering command instead of the agent's command based on a specific probability. This ensures the agent remains within "successful" states, maximizing the efficiency of the training process and preventing the agent from spending excessive time in failure states (e.g., crashing).

2.2 Reward Function and Cyclical Learning Rates

The reward function was redesigned to incorporate an imitation penalty. In addition to standard lane-keeping rewards, the agent is penalized if its steering output deviates significantly from the expert MPC-PID command ($c_{diff}$). This weight is adaptive: early in training, the agent is incentivized to mimic the expert; as performance improves, the focus shifts toward optimizing track-specific performance. To further enhance training stability and avoid local optima, the authors utilized a cyclical learning rate strategy, which oscillates the step size to find the best parameters more efficiently.

3. Experimental Setup

The system was developed and tested using the CARLA simulator, an industry-standard for autonomous driving research. Training was conducted on custom maps featuring diverse geometries, including straight sections and sharp turns. For evaluation, the researchers used two distinct environments:

  1. Circular Track: A standardized circuit with a 50-meter radius to measure low-speed precision.
  2. Digital Twin: A high-fidelity model of the Autodromo Nazionale Monza racing circuit to test the agent’s ability to generalize to unseen, high-speed environments.

4. Results and Analysis

The hybrid DRL agent demonstrated significant improvements over traditional control methods across several key metrics.

4.1 Computational Efficiency

One of the most striking results was the reduction in computational overhead. The DRL agent completed the test track in 145.28 seconds, compared to 664.23 seconds for the MPC-PID controller—a 78.13% improvement. This efficiency stems from the fact that once trained, the DRL agent functions as a feed-forward neural network, bypassing the complex real-time optimization required by MPC.

4.2 Precision and Robustness

On the circular test circuit, the agent achieved an Average Lateral Error (ALE) of only 0.00283 meters. Furthermore, the agent exhibited remarkable robustness to sensor noise. When random "jitter" was introduced to the input waypoints to simulate sensor errors, the DRL agent maintained a stable trajectory, whereas traditional controllers often struggle with such inconsistent inputs.

4.3 Generalization

The agent successfully navigated the Monza racing track without prior exposure to the environment. This indicates that the RLfD process allowed the agent to learn fundamental driving principles and predictive steering behaviors—such as smoothing out corners by observing future waypoints—rather than simply memorizing training paths.

5. Conclusion and Future Work

This study demonstrates that Reinforcement Learning from Demonstration can bridge the gap between traditional control theory and modern machine learning. By using an MPC-PID controller as a temporary "teacher," the authors produced a DRL agent that is faster, more robust, and highly precise.

Future research will focus on integrating longitudinal control (speed) with lateral control into a single unified DRL framework. Additionally, the authors aim to explore "Sim-to-Real" transfer to validate these findings on physical vehicle hardware under varying real-world weather and lighting conditions.

Summary updated

Abstract

The controller is one of the most important modules in the autonomous driving pipeline, ensuring the vehicle reaches its desired position. In this work, a reinforcement learning based lateral control approach, despite the imperfections in the vehicle models due to measurement errors and simplifications, is presented. Our approach ensures comfortable, efficient, and robust control performance considering the interface between controlling and other modules. The controller consists of the conventional Model Predictive Control (MPC)-PID part as the basis and the demonstrator, and the Deep Reinforcement Learning (DRL) part which leverages the online information from the MPC-PID part. The controller's performance is evaluated in CARLA using the ground truth of the waypoints as inputs. Experimental results demonstrate the effectiveness of the controller when vehicle information is incomplete, and the training of DRL can be stabilized with the demonstration part. These findings highlight the potential to reduce development and integration efforts for autonomous driving pipelines in the future.
BibTeX
@misc{wu2025autonomous,
  author = {Chengdong Wu and Sven Kirchner and Nils Purschke and Alois Knoll},
  title = {{Autonomous Vehicle Lateral Control Using Deep Reinforcement Learning with MPC-PID Demonstration}},
  year = {2025},
  eprint = {2506.04040},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2506.04040},
}
Figure