Epigraph Form Reinforcement Learning-based Safe Trajectory Tracking Control for Connected and Autonomous Vehicles

Xiaoke Yang, Jin Chen, Yafei Wang, Qi Zhang, Ming Li, Haiwen Wu, Chengliang Yin, Yong ZhangarXiv 2026 · arXiv

TL;DR

This paper introduces STTC-EFRL, a novel reinforcement learning framework that solves the safe trajectory tracking problem for autonomous vehicles by reformulating it into an "epigraph form." This approach eliminates the training instability common in traditional safety-constrained RL while achieving superior tracking accuracy and collision avoidance in complex, dynamic environments.

What Problem Does This Paper Address?

Safe Trajectory Tracking (STT) is the "bread and butter" of autonomous driving: a vehicle must follow a pre-planned path and speed profile precisely while ensuring it never hits anything. Traditionally, this is handled by a hierarchical stack (planner then controller), but this often leads to "module mismatch"—the planner might ask for a maneuver the car's physics simply can't execute, leading to lag or safety violations.

Deep Reinforcement Learning (DRL) offers a way to integrate planning and control into a single policy. However, standard "safe DRL" methods usually rely on Lagrangian multipliers (adaptive penalty weights). These are notoriously unstable; if a vehicle violates a safety constraint, the multiplier can explode, causing massive policy gradients that "break" the neural network's training.

Methods

The authors propose STT Control Epigraph Form RL (STTC-EFRL). Instead of using a weighted sum of tracking error and safety violations, they reformulate the problem as a min-max optimization using an auxiliary "epigraph" variable.

Key technical components include:

  • Epigraph Reformulation: The tracking cost is bounded by a scalar variable $z_k$. The objective becomes minimizing this budget while ensuring safety constraints are met. This decouples the safety "pressure" from the gradient magnitude, leading to much smoother training.
  • Two-Stage Optimization: The system solves an unconstrained inner problem (via DRL) to find the best policy for a given budget, and an outer problem (via bisection) to find the optimal budget.
  • Efficient Ray-Casting for Safety: To quantify safety, the authors developed a deterministic ray-casting algorithm to calculate a "polyhedron scaling factor" between the vehicle and obstacles. This method is significantly faster than standard iterative solvers like the Interior Point Method (IPM) and is better suited for parallel GPU training.
  • Vehicle Model: They utilize a high-fidelity 3-degree-of-freedom (3-DOF) bicycle model that captures both longitudinal and lateral dynamics.
Fig. 4. STTC-EFRL framework architecture. The solid lines denote the information flow during evaluation and implementation, and the dashed lines denote the information flow during training.

Datasets

The researchers evaluated their method using custom-designed, high-fidelity simulation and hardware-in-the-loop environments:

  • Simulated Scenarios: Three types of challenging environments were used: lane-changing (Type-1), overtaking (Type-2), and unsignalized intersections (Type-3).
  • Vehicle-in-the-Loop (ViL): A real-world validation platform consisting of an NVIDIA Jetson Orin (for on-board inference) and a chassis dynamometer to simulate physical vehicle responses.

Key Insights and Ideas

  • Gradient Stability: The "aha moment" is that by using the epigraph form, the safety constraint doesn't act as a multiplier on the gradient. Instead, the epigraph variable $z_k$ acts as a "regulator" or budget. This prevents the catastrophic oscillations seen in Lagrangian methods when safety is breached.
  • Integrated vs. Hierarchical: The paper demonstrates that an integrated controller can better follow a reference trajectory by considering safety and dynamics simultaneously, whereas hierarchical controllers often deviate conservatively.
  • Computational Efficiency: By replacing general-purpose optimization solvers with a specialized ray-casting algorithm for collision detection, the authors reduced the amortized time per safety evaluation from 76.3 $\mu$s to 15.3 $\mu$s, a 5x speedup crucial for large-scale DRL training.
Comparison showing how an integrated controller (b) maintains better trajectory tracking than a hierarchical one (a) during obstacle avoidance.

Key Results

  • Training Stability: STTC-EFRL maintained stable policy gradient norms and critic losses throughout 100,000 iterations, while Lagrangian baselines (PPO-L, DDPG-L) suffered from explosive gradients and failed to converge smoothly.
  • Tracking Performance: In Vehicle-in-the-Loop tests, STTC-EFRL achieved the lowest lateral errors across multiple scenarios. For example, in a complex lane-change scenario, it achieved a mean lateral error of -0.59m, significantly better than PPO-Lagrangian's -1.79m.
  • Safety Compliance: Unlike the non-learning baseline (UFTSTC) and some Lagrangian variants, STTC-EFRL successfully avoided collisions and respected road boundaries in all six evaluated test scenarios.

Limitations and Future Work

  • Ideal Communication: The current work assumes perfect V2X (Vehicle-to-Everything) communication. Real-world packet loss or delays could compromise safety. Future work aims to build "communication-robust" policies.
  • Model Reliance: The safety quantification currently relies on a nominal dynamic model. The authors plan to integrate Control Barrier Functions (CBFs) to provide formal mathematical safety guarantees even in the presence of unmodeled noise or disturbances.
  • Statistical Breadth: The study relied on a single training seed. Future iterations will include multi-seed training and more repeated trials to provide a more rigorous statistical profile of the algorithm's consistency.

Summary updated

Abstract

Safe trajectory tracking (STT) aims to enable autonomous vehicles to follow a reference trajectory accurately while avoiding collisions. This problem is particularly challenging for Connected and Autonomous Vehicles (CAVs) in highly dynamic environments. Existing non-learning-based STT methods often use simplified vehicle and interaction models to meet real-time requirements, which may result in suboptimal tracking accuracy. Although Deep Reinforcement Learning (DRL) can help address this limitation, widely used Lagrangian DRL methods often suffer from unstable training because increasing Lagrange multipliers amplify policy-gradient magnitudes. To address these issues, we propose a novel reinforcement learning framework for STT, called STT Control Epigraph Form RL (STTC-EFRL). In this framework, the STT problem is formulated as an epigraph-form min-max optimization problem. Based on this formulation, we develop a learning mechanism that uses a decreasing scalar related to the epigraph to adjust the policy gradient descent direction without increasing the gradient magnitude, thereby improving training stability. We also develop a computationally efficient ray-casting method to compute a polyhedron scaling factor for safety quantification. Experimental results, including Vehicle-in-the-Loop tests, show that STTC-EFRL achieves higher tracking accuracy than the considered non-learning-based baseline and more stable training than widely used Lagrangian DRL methods, while satisfying safety requirements in connected environments.
BibTeX
@misc{yang2026epigraph,
  author = {Xiaoke Yang and Jin Chen and Yafei Wang and Qi Zhang and Ming Li and Haiwen Wu and Chengliang Yin and Yong Zhang},
  title = {{Epigraph Form Reinforcement Learning-based Safe Trajectory Tracking Control for Connected and Autonomous Vehicles}},
  year = {2026},
  eprint = {2611.14903},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.14903},
}
Figure