Expert Knowledge-driven Reinforcement Learning for Autonomous Racing via Trajectory Guidance and Dynamics Constraints

Bo Leng, Weiqi Zhang, Zhuoren Li, Lu Xiong, Guizhe Jin, Ran Yu, Chen Lv

Abstract

Reinforcement learning has demonstrated significant potential in the field of autonomous driving. However, it suffers from defects such as training instability and unsafe action outputs when faced with autonomous racing environments characterized by high dynamics and strong nonlinearities. To this end, this paper proposes a trajectory guidance and dynamics constraints Reinforcement Learning (TraD-RL) method for autonomous racing. The key features of this method are as follows: 1) leveraging the prior expert racing line to construct an augmented state representation and facilitate reward shaping, thereby integrating domain knowledge to stabilize early-stage policy learning; 2) embedding explicit vehicle dynamic priors into a safe operating envelope formulated via control barrier functions to enable safety-constrained learning; and 3) adopting a multi-stage curriculum learning strategy that shifts from expert-guided learning to autonomous exploration, allowing the learned policy to surpass expert-level performance. The proposed method is evaluated in a high-fidelity simulation environment modeled after the Tempelhof Airport Street Circuit. Experimental results demonstrate that TraD-RL effectively improves both lap speed and driving stability of the autonomous racing vehicle, achieving a synergistic optimization of racing performance and safety.

Introduction

Autonomous racing represents a pinnacle of autonomous driving research, requiring vehicles to operate at the absolute limits of their physical capabilities. While Reinforcement Learning (RL) offers a path toward superhuman performance, it is often hampered by training instability and safety concerns. In high-speed scenarios, tire grip is non-linear, and minor errors can lead to catastrophic failures. Standard RL agents frequently struggle with sparse rewards and inefficient exploration, often crashing repeatedly before learning basic maneuvers. Conversely, traditional methods like Model Predictive Control (MPC) can be overly conservative and computationally intensive in complex, "at-the-limit" physics environments. This research introduces TraD-RL, a framework designed to bridge this gap by combining expert trajectory knowledge with physics-based safety constraints.

The TraD-RL Framework

The TraD-RL framework integrates three core components to stabilize learning and ensure safety: Trajectory Prior Guidance, Dynamics Constraints via Control Barrier Functions (CBFs), and a Two-Stage Curriculum Learning strategy.

Overview of the TraD-RL framework, illustrating the integration of trajectory guidance, dynamics constraints, and curriculum learning.

Trajectory Prior Guidance and State Representation

Rather than initiating learning from a blank slate, TraD-RL utilizes the Minimum Curvature Racing Line (MCRL) as a prior. This expert knowledge is integrated into the agent's state representation and reward structure. The agent's "vision" is augmented with grid-based observations of the track and its own position relative to the racing line, facilitating rapid convergence toward a sensible driving path.

Grid-based representation of track and ego observations used to augment the agent's state for more effective learning.

Dynamics Constraints via CBFs

To prevent the vehicle from spinning out or losing control, the framework embeds vehicle dynamics priors into the learning process using Control Barrier Functions (CBFs). By defining a "safe operating envelope" based on yaw rate and sideslip angle, the system applies penalties via Lagrangian Relaxation if the agent's actions threaten to exceed physical limits. A ReLU-based truncation mechanism ensures the agent is only penalized for exceeding the safety limit, avoiding the over-conservatism common in safety-constrained systems.

Two-Stage Curriculum Learning

The training process follows a structured curriculum:

  1. Stage 1 (Guidance): The agent is rewarded for following the expert MCRL at a steady, controlled speed.
  2. Stage 2 (Exploration): Speed constraints are removed, encouraging the agent to explore the state space and find faster entries and exits that may deviate from the original expert line to optimize lap times.

Experimental Results

The proposed method was evaluated on a high-fidelity simulation of the Berlin Tempelhof Airport Street Circuit (2.469 km, 17 turns) using a dynamic bicycle model with nonlinear tire forces. TraD-RL was compared against standard RL algorithms (PPO, DDPG) and the trajectory-aided method (TAL).

  • Lap Performance: TraD-RL achieved a lap time of 58.83s, significantly outperforming DDPG (75.65s) and PPO (84.67s), and surpassing the state-of-the-art TAL (61.31s).
  • Speed and Stability: The agent maintained an average speed of 39.79 m/s, a 30.5% improvement over DDPG. In complex S-curve maneuvers, TraD-RL exhibited smoother curvature transitions and fewer oscillations compared to baseline methods, which often suffered from zigzag deflections near the limits.
  • Ablation Study: Removing Trajectory Guidance (w/o TG) resulted in a dramatic drop in average speed to 21.47 m/s, confirming that expert priors are essential for efficient high-speed learning.

Conclusion

TraD-RL demonstrates that integrating expert domain knowledge with explicit physical constraints can overcome the traditional hurdles of RL in high-stakes environments. By using expert trajectories as a catalyst rather than a ceiling, the agent eventually learns to surpass expert-level performance. Future work will focus on bridging the sim-to-real gap by introducing sensor noise and varying track conditions, as well as extending the framework to multi-agent racing scenarios involving overtaking and defensive maneuvers.

Summary updated

Abstract

Reinforcement learning has demonstrated significant potential in the field of autonomous driving. However, it suffers from defects such as training instability and unsafe action outputs when faced with autonomous racing environments characterized by high dynamics and strong nonlinearities. To this end, this paper proposes a trajectory guidance and dynamics constraints Reinforcement Learning (TraD-RL) method for autonomous racing. The key features of this method are as follows: 1) leveraging the prior expert racing line to construct an augmented state representation and facilitate reward shaping, thereby integrating domain knowledge to stabilize early-stage policy learning; 2) embedding explicit vehicle dynamic priors into a safe operating envelope formulated via control barrier functions to enable safety-constrained learning; and 3) adopting a multi-stage curriculum learning strategy that shifts from expert-guided learning to autonomous exploration, allowing the learned policy to surpass expert-level performance. The proposed method is evaluated in a high-fidelity simulation environment modeled after the Tempelhof Airport Street Circuit. Experimental results demonstrate that TraD-RL effectively improves both lap speed and driving stability of the autonomous racing vehicle, achieving a synergistic optimization of racing performance and safety.
BibTeX
@misc{leng2026expert,
  author = {Bo Leng and Weiqi Zhang and Zhuoren Li and Lu Xiong and Guizhe Jin and Ran Yu and Chen Lv},
  title = {{Expert Knowledge-driven Reinforcement Learning for Autonomous Racing via Trajectory Guidance and Dynamics Constraints}},
  year = {2026},
  eprint = {2603.05842},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2603.05842},
}
Figure