Found-RL: foundation model-enhanced reinforcement learning for autonomous driving

Yansong Qu, Zihao Sheng, Zilin Huang, Jiancong Chen, Yuhao Luo, Tianyi Wang, Yiheng Feng, Samuel Labi, Sikai ChenarXiv 2026 · OpenReview

TL;DR

This paper introduces Found-RL, a novel platform that integrates large foundation models (specifically Vision-Language Models or VLMs) into reinforcement learning (RL) for autonomous driving. It tackles the sample inefficiency and lack of interpretability in traditional RL by leveraging VLMs for richer, context-aware supervision, achieving near-VLM performance with a significantly smaller, real-time capable RL model.

What Problem Does This Paper Address?

Autonomous driving (AD) systems using Reinforcement Learning (RL) have shown promise but struggle with sample inefficiency and a lack of semantic interpretability in complex scenarios. While Foundation Models (like VLMs) offer rich, context-aware knowledge that could enhance RL, integrating these computationally intensive models into high-frequency RL training loops is hindered by prohibitive inference latency and the absence of unified platforms. This paper aims to bridge this gap by providing a specialized platform that efficiently leverages foundation models to enhance RL for AD.

Methods

The core innovation is Found-RL, a platform designed for foundation model-enhanced RL in autonomous driving. Key components include:

  • Asynchronous Batch Inference Framework: This decouples heavy VLM reasoning from the real-time simulation loop, mitigating latency bottlenecks and enabling real-time or near-real-time RL training with online VLM feedback.
  • VLM Action Guidance Mechanisms:
    • Value-Margin Regularization (VMR): Encourages the RL critic to assign higher values to VLM-proposed actions than to the agent's own actions, guiding exploration.
  • Advantage-Weighted Action Guidance (AWAG): Adapts the influence of VLM guidance based on the advantage of VLM actions, promoting smoother learning.
  • CLIP-based Reward Shaping: Uses Conditional Contrastive Action Alignment to discretize vehicle speed and navigation goals, creating context-aware prompts for CLIP. This generates a normalized, margin-based reward bonus to provide dense supervision, addressing CLIP's limitations in perceiving continuous numerical values.

Datasets

The paper mentions collecting demonstration data from three benchmarks for fine-tuning VLMs:

  • CARLA Leaderboard benchmark
  • NoCrash benchmark
  • CARLA Challenge benchmark

Key Insights and Ideas

  • VLMs as "Tireless Mentors": Foundation models can act as scalable, always-available mentors, providing semantic reasoning capabilities that bridge the gap between imitation learning and RL.
  • Asynchronous Inference is Crucial: Decoupling VLM inference from the simulation loop is essential for real-time performance and practical application of VLM feedback in RL.
  • Lightweight RL + VLM Supervision > Heavy VLM: A compact RL model augmented with VLM feedback can achieve performance comparable to, or even exceeding, much larger VLM-only models, especially in terms of efficiency and safety.
  • Semantic Supervision for Safety: VLM feedback, particularly through reward shaping and action guidance, can effectively instill safety-critical behaviors and improve rule compliance.
  • Discretization for CLIP: Discretizing continuous states (like speed) and actions allows CLIP to be effectively used for reward shaping in dynamic driving scenarios.

Key Results

  • Found-RL, with a lightweight RL model (3.82M parameters), achieves 500 FPS, a significant speedup compared to VLMs (around 1 FPS).
  • DrQv2-CLIP achieved a Driving Score of 0.77 on the Leaderboard benchmark, outperforming baselines and even larger VLMs like Qwen2.5-vl-7B (0.76).
  • VLM-enhanced agents demonstrated reduced collision rates (e.g., SAC-VMR lowered vehicle collisions from 0.50 to 0.20 per km).
  • The methods maintained high energy efficiency, with SAC-VMR achieving the lowest battery consumption (Icell = 0.07).
  • On the NoCrash benchmark, DrQv2-CLIP achieved a Success Rate of 0.71 in the unseen Town02, a significant improvement from a failing 7%.
  • Found-RL agents showed superior adherence to static traffic rules, with significantly lower red light violation rates compared to VLMs.

Limitations and Future Work

  • Inference Acceleration and Quantization: Future work will explore inference acceleration and quantization to prevent stale feedback and ensure broad sample coverage when scaling to larger models.
  • Challenging Benchmarks: The authors aim to validate robustness on more challenging benchmarks like CARLA Leaderboard 2.0.
  • Multi-VLM Ensembles: They envision orchestrating multi-VLM ensembles where distinct models provide specialized feedback to collaboratively enhance learning.
  • Lightweight VLMs as Policy Backbones: Integrating lightweight VLMs directly as policy backbones for end-to-end planning is another future direction.

Summary updated

BibTeX
@misc{qu2026found,
  author = {Yansong Qu and Zihao Sheng and Zilin Huang and Jiancong Chen and Yuhao Luo and Tianyi Wang and Yiheng Feng and Samuel Labi and Sikai Chen},
  title = {{Found-RL: foundation model-enhanced reinforcement learning for autonomous driving}},
  year = {2026},
  url = {https://openreview.net/forum?id=Ag5yNF0mlH},
}
Figure