TL;DR
This paper introduces Found-RL, a novel platform that integrates large foundation models (specifically Vision-Language Models or VLMs) into reinforcement learning (RL) for autonomous driving. It tackles the sample inefficiency and lack of interpretability in traditional RL by leveraging VLMs for richer, context-aware supervision, achieving near-VLM performance with a significantly smaller, real-time capable RL model.
What Problem Does This Paper Address?
Autonomous driving (AD) systems using Reinforcement Learning (RL) have shown promise but struggle with sample inefficiency and a lack of semantic interpretability in complex scenarios. While Foundation Models (like VLMs) offer rich, context-aware knowledge that could enhance RL, integrating these computationally intensive models into high-frequency RL training loops is hindered by prohibitive inference latency and the absence of unified platforms. This paper aims to bridge this gap by providing a specialized platform that efficiently leverages foundation models to enhance RL for AD.
Methods
The core innovation is Found-RL, a platform designed for foundation model-enhanced RL in autonomous driving. Key components include:
- Asynchronous Batch Inference Framework: This decouples heavy VLM reasoning from the real-time simulation loop, mitigating latency bottlenecks and enabling real-time or near-real-time RL training with online VLM feedback.
- VLM Action Guidance Mechanisms:
- Value-Margin Regularization (VMR): Encourages the RL critic to assign higher values to VLM-proposed actions than to the agent's own actions, guiding exploration.
- Advantage-Weighted Action Guidance (AWAG): Adapts the influence of VLM guidance based on the advantage of VLM actions, promoting smoother learning.
- CLIP-based Reward Shaping: Uses Conditional Contrastive Action Alignment to discretize vehicle speed and navigation goals, creating context-aware prompts for CLIP. This generates a normalized, margin-based reward bonus to provide dense supervision, addressing CLIP's limitations in perceiving continuous numerical values.
Datasets
The paper mentions collecting demonstration data from three benchmarks for fine-tuning VLMs:
- CARLA Leaderboard benchmark
- NoCrash benchmark
- CARLA Challenge benchmark
Key Insights and Ideas
- VLMs as "Tireless Mentors": Foundation models can act as scalable, always-available mentors, providing semantic reasoning capabilities that bridge the gap between imitation learning and RL.
- Asynchronous Inference is Crucial: Decoupling VLM inference from the simulation loop is essential for real-time performance and practical application of VLM feedback in RL.
- Lightweight RL + VLM Supervision > Heavy VLM: A compact RL model augmented with VLM feedback can achieve performance comparable to, or even exceeding, much larger VLM-only models, especially in terms of efficiency and safety.
- Semantic Supervision for Safety: VLM feedback, particularly through reward shaping and action guidance, can effectively instill safety-critical behaviors and improve rule compliance.
- Discretization for CLIP: Discretizing continuous states (like speed) and actions allows CLIP to be effectively used for reward shaping in dynamic driving scenarios.
Key Results
- Found-RL, with a lightweight RL model (3.82M parameters), achieves 500 FPS, a significant speedup compared to VLMs (around 1 FPS).
- DrQv2-CLIP achieved a Driving Score of 0.77 on the Leaderboard benchmark, outperforming baselines and even larger VLMs like Qwen2.5-vl-7B (0.76).
- VLM-enhanced agents demonstrated reduced collision rates (e.g., SAC-VMR lowered vehicle collisions from 0.50 to 0.20 per km).
- The methods maintained high energy efficiency, with SAC-VMR achieving the lowest battery consumption (Icell = 0.07).
- On the NoCrash benchmark, DrQv2-CLIP achieved a Success Rate of 0.71 in the unseen Town02, a significant improvement from a failing 7%.
- Found-RL agents showed superior adherence to static traffic rules, with significantly lower red light violation rates compared to VLMs.
Limitations and Future Work
- Inference Acceleration and Quantization: Future work will explore inference acceleration and quantization to prevent stale feedback and ensure broad sample coverage when scaling to larger models.
- Challenging Benchmarks: The authors aim to validate robustness on more challenging benchmarks like CARLA Leaderboard 2.0.
- Multi-VLM Ensembles: They envision orchestrating multi-VLM ensembles where distinct models provide specialized feedback to collaboratively enhance learning.
- Lightweight VLMs as Policy Backbones: Integrating lightweight VLMs directly as policy backbones for end-to-end planning is another future direction.
Summary updated