Learning Autonomous Race Driving with Action Mapping Reinforcement Learning

Yuanda Wang, Xin Yuan, Changyin SunarXiv 2024 · DOI · arXiv · OpenReview

TL;DR

This paper introduces a novel "Action Mapping Reinforcement Learning" (AM-RL) approach for autonomous race driving. By mapping unconstrained actions to safe, friction-limited control inputs, AM-RL allows race cars to operate at their physical limits, achieving significantly faster lap times and higher success rates than conventional RL methods.

What Problem Does This Paper Address?

Autonomous race driving is a high-stakes challenge. Unlike typical driving scenarios where vehicles operate near equilibrium, race cars push the limits of tire-road friction. This creates complex, state-dependent constraints on control inputs (steering, acceleration, braking) that are difficult to manage. Traditional reinforcement learning (RL) methods often resort to conservative policies or struggle to guarantee safety when operating near these physical limits, leading to slower lap times or even crashes. The core problem is how to train an RL agent that can safely and aggressively drive a race car at its performance edge, respecting dynamic friction constraints.

Methods

The paper's key innovation is the Action Mapping (AM) mechanism integrated into an RL framework. Instead of directly outputting control signals, the RL agent (specifically, a neural network policy) outputs "virtual actions." These virtual actions are then transformed into real control inputs (throttle/brake and steering) by the AM mechanism. This mapping is designed to ensure that the resulting control inputs always respect the tire-road friction limits, which vary with the car's speed and steering angle.

To handle the complex, non-linear nature of these friction constraints, the authors propose a numerical approximation method for implementing the AM. This involves:

  1. Discretizing the state space (speed, steering angle) and action space.
  2. Sampling the vehicle's dynamic model to determine the boundary of the admissible control input space for various states.
  3. Building a lookup table of the maximum allowable control input magnitude for each direction.
  4. Using multidimensional interpolation to approximate this boundary function for continuous states.
  5. When a virtual action is generated, the AM maps it to a real control input that has the same direction but is scaled to fit within the calculated safe boundary.

This AM mechanism is integrated with the Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm, a state-of-the-art off-policy RL algorithm known for its stability in continuous control tasks. The TD3 algorithm is modified to use the AM for generating real control inputs.

Datasets

The authors developed their own race simulator for training and evaluation. They created two distinct race tracks:

  • Track-A: A simpler track with five corners.
  • Track-B: A more complex track modeled after the Ruisi Circuit in China, featuring ten corners and narrower width, posing a greater challenge.

No external datasets were used for training or evaluation.

Key Insights and Ideas

  • Action Mapping as a Constraint Handler: AM provides a principled way to enforce state-dependent constraints by transforming actions after the policy generates them, rather than trying to bake constraints into the reward function or policy output directly. This avoids the conservatism often associated with reward shaping for safety.
  • Numerical AM for Complex Dynamics: The proposed numerical approximation method for AM is crucial. It allows the system to handle the intricate, non-linear relationship between vehicle state, control inputs, and friction limits without requiring an explicit, closed-form analytical solution, which is often intractable.
  • Generalization to Friction Conditions: A key advantage of AM is its ability to adapt to varying friction conditions. By simply adjusting the friction limit parameter in the AM function, the same learned policy can be made to operate safely under different grip levels, demonstrating generalization without full retraining.
  • Pushing the Performance Envelope: By ensuring safety through AM, the RL agent can be trained to operate much closer to the vehicle's physical limits, leading to significantly improved lap times.

Key Results

  • Superior Lap Times: The TD3-AM approach achieved the best lap times on both Track-A (22% shorter than baseline TD3) and Track-B (1:04.16, outperforming other policies).
  • High Success Rate: TD3-AM achieved a 90% success rate (completing at least two laps without critical errors) on Track-A, significantly higher than other methods (all below 50%).
  • Zero Constraint Violations: The AM mechanism successfully avoided any tire friction constraint violations during evaluation, a feat not achieved by other methods, including the safety layer (SL) technique.
  • Generalization Demonstrated: The TD3-AM policy could adapt to different friction constraints (1.15g vs. 1.0g) by adjusting the AM function, maintaining safety and achieving comparable, albeit slower, lap times under reduced grip.
  • Smoother Control: Trajectories and control signals from TD3-AM were observed to be smoother and more consistent than baseline TD3, indicating better vehicle handling.

Limitations and Future Work

  • Simulation-Based: The current evaluation is performed in a simulator. The authors plan to assess the framework's applicability in real-world race experiments.
  • Conservative Friction Constraint: For real-world deployment, they suggest starting with a conservative friction constraint function and progressively refining it through online learning due to the difficulty of obtaining an accurate dynamic model.
  • Sensor Noise and Uncertainties: Future work will explore methods to handle sensor noise and other uncertainties in real-world applications.
  • Broader Applications: The authors aim to investigate AM-RL's capabilities in addressing other nonlinear control problems with constraints.

Summary updated

Abstract

Autonomous race driving poses a complex control challenge as vehicles must be operated at the edge of their handling limits to reduce lap times while respecting physical and safety constraints. This paper presents a novel reinforcement learning (RL)-based approach, incorporating the action mapping (AM) mechanism to manage state-dependent input constraints arising from limited tire-road friction. A numerical approximation method is proposed to implement AM, addressing the complex dynamics associated with the friction constraints. The AM mechanism also allows the learned driving policy to be generalized to different friction conditions. Experimental results in our developed race simulator demonstrate that the proposed AM-RL approach achieves superior lap times and better success rates compared to the conventional RL-based approaches. The generalization capability of driving policy with AM is also validated in the experiments.
BibTeX
@misc{wang2024learning,
  author = {Yuanda Wang and Xin Yuan and Changyin Sun},
  title = {{Learning Autonomous Race Driving with Action Mapping Reinforcement Learning}},
  year = {2024},
  eprint = {2406.14934},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2406.14934},
}
Figure