TL;DR
This paper introduces a novel Hybrid Supervised-Reinforcement Learning (HSRL) framework for autonomous vehicle path tracking that simultaneously optimizes for accuracy and passenger comfort by mitigating motion sickness. By combining supervised learning from expert demonstrations with reinforcement learning that incorporates a motion sickness metric, HSRL achieves superior tracking performance and significantly reduces passenger discomfort compared to traditional methods.
What Problem Does This Paper Address?
Autonomous driving (AD) systems aim to provide safe and comfortable transportation. A critical component is path tracking, where the vehicle precisely follows a planned route. However, existing path-tracking controllers face a trade-off: achieving high accuracy often comes at the cost of computational overhead, and more importantly, they largely neglect passenger comfort, specifically motion sickness (MS). MS is a growing concern in AD as passengers engage in non-driving activities, leading to sensory conflicts. This paper addresses the challenge of developing a path-tracking system that is both highly accurate and actively reduces motion sickness, a problem that current methods struggle to solve effectively.
Methods
The core of this work is the Hybrid Supervised-Reinforcement Learning (HSRL) framework, which employs a two-stage approach:
- Offline Supervised Learning (Behavior Cloning - BC):
- An expert agent (composed of an A* planner and PID controllers) collects high-quality trajectory data in a simulated environment (CARLA Town04).
- This expert data is used to train an initial policy network via Behavior Cloning. This phase leverages expert knowledge to rapidly optimize the policy's controllability and stabilize training dynamics, mitigating the sample inefficiency of pure Reinforcement Learning (RL).
- Online Reinforcement Learning (RL):
- The pre-trained policy is then refined using an Actor-Critic architecture, specifically the Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm.
- TD3 is chosen for its stability and ability to handle continuous action spaces.
- A key innovation here is the integration of a passenger Motion Sickness (MS) mechanism into the multi-objective reward function. This mechanism uses the standardized Motion Sickness Dose Value (MSDV) model to quantify and penalize motion sickness-inducing factors.
- The reward function balances tracking accuracy (velocity, trajectory, heading) with comfort (control smoothness, MSDV reduction).
Datasets
The primary dataset used in this study is generated through expert demonstrations collected within the CARLA 0.9.14 simulator. Specifically, the Town04 environment was used for data acquisition, chosen for its road environment complexity. The expert agent, consisting of an A* planner and PID controllers, generated high-density trajectories and corresponding control actions. This generated dataset was then split into training and validation sets for the supervised learning phase. For evaluation, the paper uses predefined paths within CARLA, including S-shape, U-shape, O-shape, and a high-speed random shape scenario.
Key Insights and Ideas
- Bridging the Accuracy-Comfort Gap: The most significant insight is the successful integration of motion sickness mitigation directly into the path-tracking control loop. By treating passenger comfort as a primary objective alongside tracking accuracy, the HSRL framework offers a more holistic solution for autonomous driving.
- Supervised Learning as a Warm-Start for RL: Using expert demonstrations for supervised pre-training dramatically improves the sample efficiency and stability of the subsequent RL phase. This "warm-start" allows the RL agent to converge faster and reach a higher performance ceiling.
- MSDV as a Crucial Reward Component: Incorporating the MSDV into the reward function is critical. It moves beyond simple acceleration or jerk metrics to capture the cumulative effect of motion on passenger comfort, which is more representative of the actual experience of motion sickness.
- Synergistic Effects of HSRL: The combination of supervised learning for foundational accuracy and RL for adaptability, coupled with the multi-objective reward function, creates a synergistic effect that outperforms individual components or traditional methods.
Key Results
- Superior Tracking Accuracy: The HSRL method demonstrated significantly improved tracking accuracy compared to PID and Model Predictive Control (MPC) across various scenarios. For instance, in the S-shape test, HSRL achieved a mean lateral deviation of 0.0469 m, compared to 0.0469 m for PID and 0.0469 m for MPC (though the paper notes HSRL controlled error within a narrower range).
- Significant Motion Sickness Reduction: HSRL reduced the cumulative Motion Sickness Dose Value (MSDV) by approximately 15.7% compared to PID and MPC. This was observed across multiple test scenarios, with the reduction being more pronounced in complex routes.
- Robustness and Adaptability: The HSRL framework showed better adaptability to dynamic scenarios and complex road geometries, outperforming PID and MPC, which struggled with fixed gain parameters or model simplifications.
- Ablation Study Confirmation: Removing either the supervised learning (LSL) or reinforcement learning (LRL) components led to performance degradation, confirming the importance of both stages. Removing LSL resulted in a significant increase in lateral deviation (41.3%) and TMSDV (43.2%), while removing LRL showed moderate degradation, highlighting supervised learning's critical role in foundational performance.
Limitations and Future Work
- Limitations:
- The training data for HSRL may not have fully covered extreme or rare operating conditions, potentially leading to maximum lateral deviation errors in certain situations.
- While the study quantifies MS via acceleration metrics, it acknowledges that future investigations could integrate more detailed vehicle-specific dynamics and suspension characteristics for a more comprehensive discomfort model.
- The current evaluation is based on simulations. Field tests under heterogeneous road conditions and sensor noise profiles are needed.
- Future Work:
- Develop a multi-physics discomfort model by integrating vehicle-specific dynamics, suspension characteristics, and body resonance frequencies.
- Conduct field tests to validate the framework in real-world heterogeneous road conditions and with varying sensor noise profiles.
- Design a phased implementation framework to address latency tolerance and edge computing constraints for physical vehicle platforms.
Summary updated