Life-Long Multi-Task Learning of Adaptive Path Tracking Policy for Autonomous Vehicle

Cheng Gong, Jianwei Gong, Chao Lu, Zhe Liu, Zirui Li

Abstract

This paper proposes a life-long adaptive path tracking policy learning method for autonomous vehicles that can self-evolve and self-adapt with multi-task knowledge. Firstly, the proposed method can learn a model-free control policy for path tracking directly from the historical driving experience, where the property of vehicle dynamics and corresponding control strategy can be learned simultaneously. Secondly, by utilizing the life-long learning method, the proposed method can learn the policy with task-incremental knowledge without encountering catastrophic forgetting. Thus, with continual multi-task knowledge learned, the policy can iteratively adapt to new tasks and improve its performance with knowledge from new tasks. Thirdly, a memory evaluation and updating method is applied to optimize memory structure for life-long learning which enables the policy to learn toward selected directions. Experiments are conducted using a high-fidelity vehicle dynamic model in a complex curvy road to evaluate the performance of the proposed method. Results show that the proposed method can effectively evolve with continual multi-task knowledge and adapt to the new environment, where the performance of the proposed method can also surpass two commonly used baseline methods after evolving.

Introduction: The Challenge of Adaptive Path Tracking

Path tracking—the ability of an autonomous vehicle (AV) to follow a specific trajectory—is a fundamental requirement for self-driving systems. While traditional controllers like PID or Model Predictive Control (MPC) perform well in stable, well-defined conditions, they often struggle when vehicle dynamics change (e.g., varying tire stiffness or payloads) or when encountering unfamiliar environments.

Path tracking geometry and variables including lateral error (e) and heading error (theta).

Machine learning, particularly reinforcement learning, offers a way to "learn" these complex dynamics from data. However, it faces two major hurdles:

  1. Online Adaptability: Real-time updates are often computationally prohibitive.
  2. Catastrophic Forgetting: Standard neural networks tend to "overwrite" previous knowledge when learning a new task (e.g., learning to drive on wet roads might cause the model to forget how to drive on dry ones).

This research introduces a model-free, life-long learning framework that allows an AV to evolve as it gains driving experience, adapting to new tasks without losing prior proficiency.

Proposed Methodology

The researchers developed a three-part framework to address these challenges:

1. Model-Free Policy Learning

Instead of relying on a rigid mathematical model, the policy learns directly from historical driving data. It treats past successful trajectories as "perfect" examples to imitate, effectively learning the relationship between the path ahead and the required steering commands ($\delta$).

2. Average Gradient Episodic Memory (AGEM)

To prevent catastrophic forgetting, the system employs an "episodic memory" of past tasks. During training on new data, the AGEM algorithm ensures that updates to the neural network's weights do not increase the error (loss) on data stored in this memory. This constrains the learning direction, ensuring new skills do not degrade old ones.

3. Memory Evaluation and Updating

To optimize the memory structure, the authors introduced a selection rule: if new data is similar to an existing memory, the system compares them and retains the one that achieved the "minimal steering effort." This heuristic ensures the memory remains lean and prioritizes driving comfort.

Experimental Setup and Datasets

The study was conducted in a high-fidelity simulation environment using MATLAB/SIMULINK and the Vehicle Dynamic Toolbox.

Experimental setup showing the full track (a), simulation view (b), and specific path segments S1-S3 (c) used for multi-task learning.
  • Vehicle Model: A dual-track model with 3 Degrees of Freedom (DOF).
  • Scenarios: A complex curved road was segmented into three sections (S1, S2, S3) with varying curvatures.
  • Data Collection: Initial data was generated using a standard MPC at various speeds (3m/s to 15m/s). The final dataset consisted of 25 to 50 distinct "tasks" based on different road sections and velocities.

Key Insights and Results

The experiments yielded several significant findings:

  • Experience Accumulation: Learning from "easy" tasks significantly improved performance on "difficult" tasks later. The accumulation of diverse experiences allowed the policy to generalize better than models trained on single scenarios.
  • Stability vs. Forgetting: While a standard "Non-Life-Long" baseline showed performance spikes (indicating forgetting) after 40 tasks, the proposed LL-ME (Life-Long with Memory Evaluation) method maintained a stable, low error rate.
  • Tracking Accuracy: In tests on scenario S1 at 10m/s, the LL-ME policy achieved a lower lateral tracking error than both Pure Pursuit (PP) and dynamic-MPC.
  • Efficiency: With an average inference time of just 1.1ms, the policy is highly suitable for real-time onboard applications.
  • Smoothness: The life-long learning models produced significantly smoother steering control signals compared to the Non-LL model, which became "jittery" as it encountered more tasks.

Conclusion and Future Work

The proposed life-long learning framework demonstrates that autonomous vehicles can effectively evolve and adapt to new environments by leveraging multi-task knowledge. By using steering effort as a metric for memory selection, the system inherently learns more comfortable driving behaviors over time.

Future work will focus on reducing pre-training requirements, exploring richer road state representations, and testing the system in unpredictable real-world environments, such as off-road terrain or dense traffic.

Summary updated

Abstract

This paper proposes a life-long adaptive path tracking policy learning method for autonomous vehicles that can self-evolve and self-adapt with multi-task knowledge. Firstly, the proposed method can learn a model-free control policy for path tracking directly from the historical driving experience, where the property of vehicle dynamics and corresponding control strategy can be learned simultaneously. Secondly, by utilizing the life-long learning method, the proposed method can learn the policy with task-incremental knowledge without encountering catastrophic forgetting. Thus, with continual multi-task knowledge learned, the policy can iteratively adapt to new tasks and improve its performance with knowledge from new tasks. Thirdly, a memory evaluation and updating method is applied to optimize memory structure for life-long learning which enables the policy to learn toward selected directions. Experiments are conducted using a high-fidelity vehicle dynamic model in a complex curvy road to evaluate the performance of the proposed method. Results show that the proposed method can effectively evolve with continual multi-task knowledge and adapt to the new environment, where the performance of the proposed method can also surpass two commonly used baseline methods after evolving.
BibTeX
@misc{gong2021life,
  author = {Cheng Gong and Jianwei Gong and Chao Lu and Zhe Liu and Zirui Li},
  title = {{Life-Long Multi-Task Learning of Adaptive Path Tracking Policy for Autonomous Vehicle}},
  year = {2021},
  eprint = {2109.07210},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2109.07210},
}
Figure