Foundation Models for Autonomous Driving System: An Initial Roadmap

TL;DR

This paper presents a comprehensive, software-engineering-focused roadmap for integrating Foundation Models (FMs)—including LLMs, VLMs, and world models—into Autonomous Driving Systems (ADSs). By analyzing the entire lifecycle across infrastructure, in-vehicle integration, and real-world deployment, the authors identify critical safety, reliability, and privacy roadblocks and outline a structured research path to overcome them.

What Problem Does This Paper Address?

Traditional autonomous driving systems rely heavily on rule-based architectures and supervised learning over finite, labeled datasets. While highly effective in controlled environments, these systems struggle with "long-tail" scenarios—rare, unexpected obstacles, extreme weather, or complex human interactions that characterize real-world driving.

Foundation Models (FMs) trained on massive, diverse datasets offer unprecedented reasoning and generalization capabilities, promising to bridge this gap. However, ADSs are safety-critical cyber-physical systems. Naively deploying complex, resource-heavy FMs into vehicles introduces severe software engineering challenges: data curation biases, privacy leaks, high on-board computational latency, hardware security vulnerabilities, and the notorious problem of model "hallucination." This paper provides a structured framework to understand how infrastructure-level decisions propagate to cause real-world system failures.

Methods

The authors conducted a structured literature review spanning publications from January 2020 to January 2026. Rather than focusing on a single model family or narrow task, they propose a three-dimensional roadmap framework:

1. FM4ADS Infrastructure: Focuses on data curation (debiasing, privacy preservation, augmentation), development toolchains, and computational resources (hardware roots of trust, Trusted Execution Environments).
2. FM in Vehicle: Examines core module enhancement (perception, planning, control, end-to-end AD) and user experience (conversational interfaces, surrounding awareness).
3. FM4ADS in Practice: Explores real-world deployment strategies, distinguishing between modular integration (using FMs for specific components) and full adoption (end-to-end FM control).

Expand

Crucially, the authors develop an End-to-End Dependency Analysis that traces root causes (e.g., unscrubbed personally identifiable information in training data) to system effects (data memorization) and final consequences (privacy leaks during inference). They also provide a risk-utility assessment framework for model optimization techniques like quantization and pruning.

Datasets

As a roadmap and literature synthesis, this paper does not introduce a new empirical dataset. Instead, it systematically categorizes the datasets and simulators driving current research:

• Simulators & Environments: CARLA, IdSim, and HighwayEnv are widely used to test decision-making, planning, and scenario generation.
• Core Datasets: Benchmarks like nuScenes, BDD-X, Talk2BEV-Bench, SUP-AD, and VIVA are evaluated based on metrics like Mean Absolute Error (MAE), Intersection over Union (IoU), and collision rates.
• Data Shift: The paper highlights a major transition from traditional 2D bounding-box datasets to multimodal-language datasets featuring rich language annotations for scene understanding.

Key Insights and Ideas

• The Cascade of Failure: Infrastructure decisions directly dictate real-world safety. For example, aggressive model optimization (quantization/pruning) implemented to meet strict on-board latency constraints can inadvertently lower robustness thresholds, making the system highly vulnerable to hardware-level or bit-flip attacks.
• Brittle Cascades vs. End-to-End Fusion: Most current in-vehicle integrations rely on a cascaded pipeline where FMs process the outputs of upstream perception modules. If the perception module makes a slight error (e.g., misjudging an object's heading), it propagates catastrophically downstream. End-to-end multimodal fusion directly from raw sensor data (camera, LiDAR, radar) is needed to allow cross-modal compensation.
• The Latency-Trustworthiness Tension: There is a strict 500ms latency threshold for real-time driving reasoning. Achieving this requires aggressive compression, which currently triggers higher rates of hallucination and adversarial vulnerability.
• Hardware Isolation Gaps: Confining an FM-scale workload to a secure Trusted Execution Environment (TEE) faces memory capacity limits and massive I/O marshalling overheads for high-bandwidth sensor streams (LiDAR/cameras).

Key Results

The paper synthesizes the state-of-the-art across industry and academia:

• Modular Integration in Practice: Highlights production deployments like Xiaomi SU7 (VLM for scene interpretation), Li Auto (VLM-enhanced scenario recognition), and TIER IV (LLM for human-vehicle interaction).
• Full Adoption: Highlights experimental frameworks like Talk2Drive (personalized end-to-end driving) and Waymo's MotionLM (multi-agent motion prediction mapped as a language task).

The authors establish concrete timelines for the research community:

• Short-Term Goals: Unified data management frameworks, automated license compliance tools, and optimizing distributed training tools (like Ray).
• Mid-Term Goals: End-to-end multimodal fusion models, robust edge-cloud task orchestration, and open-source domain-specific FMs for AD.
• Long-Term Goals: Silicon-level hardware roots of trust (TEEs for AI accelerators), formal verification of FM-generated code, and advanced AI alignment (forward/backward/online safety alignment).

Limitations and Future Work

• Limitations: As a roadmap based on a literature review, the paper does not present new empirical performance metrics or novel model architectures. It notes that current literature is heavily fragmented, often focusing on narrow tasks (like generating test scenarios) rather than systemic safety.
• Future Work: The authors call for urgent research into "Robust-by-Design" optimization techniques (compression algorithms that preserve safety/robustness), automated verification loops for LLM-generated production code, and standardized benchmarks specifically designed to evaluate hallucination and multi-modal sensor fusion under noise.