CurricuVLM: Towards safe autonomous driving via personalized safety-critical curriculum learning with vision-language models

Zihao Sheng, Zilin Huang, Yansong Qu, Yue Leng, Sruthi Chandana Bhavanam, Sikai Chen

TL;DR

Autonomous driving systems struggle with rare but critical safety scenarios. This paper introduces CurricuVLM, a novel framework that uses Vision-Language Models (VLMs) to dynamically generate personalized training curricula, adapting to an AV's learning progress and specifically targeting its weaknesses to improve safety.

What Problem Does This Paper Address?

Ensuring the safety of autonomous vehicles (AVs) is paramount, especially when encountering rare, high-consequence "safety-critical" events. While researchers can generate these scenarios, effectively integrating them into the training process to improve AV policy learning has been a challenge. Existing methods often use static sets of pre-generated scenarios, which become less effective as the AV learns and its capabilities evolve. Furthermore, creating personalized training curricula that adapt to an individual AV's specific performance bottlenecks and learning progress remains largely unexplored. This paper aims to bridge this gap by developing a system that can analyze AV behavior in safety-critical situations and dynamically generate tailored training experiences.

Methods

CurricuVLM employs a closed-loop learning system with three core mechanisms:

  1. Safety-Critical Event Analysis with Vision-Language Models (VLMs): When an AV encounters a safety-critical event during training, its visual observations are fed into a VLM (specifically GPT-40 in this work). The VLM generates detailed textual descriptions of the event, including the nature of the violation, object positioning, AV responses, and contextual factors.
  2. Behavioral Pattern Analysis and Recommendation: These VLM-generated descriptions are then processed by GPT-40, guided by a structured prompt. This analysis identifies recurring behavioral patterns, causal factors for failures, and specific areas of weakness. The output includes a "Situation," "Reasoning," "Recommendation," and "CriticalArea."
  3. Personalized Curriculum Adaptation via Scenario Generation: Based on the insights from GPT-40, the framework dynamically generates personalized safety-critical scenarios. This involves predicting background vehicle (BV) trajectories using a pre-trained model (DenseTNT) and then optimizing BV trajectories to create situations that specifically target the AV's identified weaknesses. This is formulated as a conditional trajectory generation problem.
  4. Dynamic Curriculum Scheduling: A probability-based mechanism adaptively integrates these personalized safety-critical scenarios into the training curriculum alongside regular scenarios. This ensures a progressive increase in difficulty and exposure to critical events as the AV learns.

Datasets

The experiments were conducted using the Waymo Open Motion Dataset. This dataset provides real-world driving scenarios with detailed trajectory information for ego vehicles and surrounding traffic participants, along with HD map annotations. The authors used 500 representative scenarios, splitting them into 400 for training and 100 for testing.

Key Insights and Ideas

  • VLMs as "Observant Mentors": The core innovation is using VLMs not just for scene understanding but as active participants in the learning process, akin to a human instructor analyzing mistakes and designing targeted exercises.
  • Personalization is Key: Static curricula are insufficient. Dynamically adapting training scenarios to an AV's specific failure modes and evolving capabilities is crucial for robust safety.
  • Bridging the Gap: CurricuVLM effectively bridges the gap between scenario generation and policy learning by creating a feedback loop where observed failures directly inform future training.
  • Interpretable Analysis: The VLM-based analysis provides human-understandable insights into why an AV fails, which is a significant advantage over black-box approaches.

Key Results

CurricuVLM demonstrated superior performance across various metrics compared to state-of-the-art baselines (including traditional RL, safe RL, imitation learning, and other curriculum learning methods) on both regular and safety-critical test scenarios.

  • Regular Scenarios: Achieved higher episode rewards (52.3 vs. 49.3 for TD3) and road completion rates (76.2% vs. 72.9% for TD3), while maintaining a competitive crash rate (16.0%).
  • Safety-Critical Scenarios: Showed significant improvements, with a substantially lower crash rate (25.1% vs. 39.7% for TD3), higher episode rewards (48.9 vs. 42.4 for TD3), and superior road completion rates (73.4% vs. 65.2% for TD3).
  • Learning Effectiveness: Achieved a significantly higher failure-to-success rate (39.1% vs. 28.6% for TD3), indicating better learning from mistakes.
  • Algorithm Agnostic: The framework showed compatibility with various RL algorithms (TD3, SAC, PPO), suggesting its general applicability.

Limitations and Future Work

  • VLM Choice Sensitivity: The performance can be sensitive to the choice of VLM. While GPT-40 and Llama-3.2-11B-Vision performed well, others showed limitations.
  • Batch Size Impact: The batch size for VLM analysis needs careful tuning; too small leads to conservative behavior, too large delays analysis.
  • Future Work:
    • Fine-tuning VLMs on autonomous driving data for improved understanding.
  • Incorporating Retrieval-Augmented Generation (RAG) to reference traffic rules and guidelines.
  • Developing methods to directly generate realistic scenarios from VLM natural language recommendations.

Summary updated

Abstract

Ensuring safety in autonomous driving systems remains a critical challenge, particularly in handling rare but potentially catastrophic safety-critical scenarios. While existing research has explored generating safety-critical scenarios for autonomous vehicle (AV) testing, there is limited work on effectively incorporating these scenarios into policy learning to enhance safety. Furthermore, developing training curricula that adapt to an AV's evolving behavioral patterns and performance bottlenecks remains largely unexplored. To address these challenges, we propose CurricuVLM, a novel framework that leverages Vision-Language Models (VLMs) to enable personalized curriculum learning for autonomous driving agents. Our approach uniquely exploits VLMs' multimodal understanding capabilities to analyze agent behavior, identify performance weaknesses, and dynamically generate tailored training scenarios for curriculum adaptation. Through comprehensive analysis of unsafe driving situations with narrative descriptions, CurricuVLM performs in-depth reasoning to evaluate the AV's capabilities and identify critical behavioral patterns. The framework then synthesizes customized training scenarios targeting these identified limitations, enabling effective and personalized curriculum learning. Extensive experiments on the Waymo Open Motion Dataset show that CurricuVLM outperforms state-of-the-art baselines across both regular and safety-critical scenarios, achieving superior performance in terms of navigation success, driving efficiency, and safety metrics. Further analysis reveals that CurricuVLM serves as a general approach that can be integrated with various RL algorithms to enhance autonomous driving systems. The code and demo video are available at: https://zihaosheng.github.io/CurricuVLM/.
BibTeX
@inproceedings{sheng2026curricuvlm,
  author = {Zihao Sheng and Zilin Huang and Yansong Qu and Yue Leng and Sruthi Chandana Bhavanam and Sikai Chen},
  title = {{CurricuVLM: Towards safe autonomous driving via personalized safety-critical curriculum learning with vision-language models}},
  booktitle = {Transportation},
  year = {2026},
  doi = {10.1016/j.trc.2026.105549},
}

Loading related content...

Figure