DriveGenVLM: Real-world Video Generation for Vision Language Model based Autonomous Driving

Yongjie Fu, Anmol Jain, Xu Chen, Zhaobin Mo, Xuan Di

Abstract

Autonomous driving systems require sophisticated scene understanding and the ability to predict future environmental states. DriveGenVLM introduces a novel framework that integrates diffusion-based video generation with Vision Language Models (VLMs) to enhance these capabilities. By utilizing a Flexible Diffusion Model (FDM) to generate realistic future driving sequences and the EILEV model to interpret them, this research demonstrates that AI-generated videos can provide the semantic depth necessary for high-level reasoning and decision-making in autonomous agents.

Introduction

Modern autonomous driving research is shifting from simple object detection toward comprehensive scene "understanding." To make safe decisions, an agent must anticipate how a scene will evolve over time. While Vision Language Models (VLMs) have shown significant promise in reasoning about visual data, their effectiveness is heavily dependent on the quality and coherence of the video inputs they receive.

Generating realistic, long-term driving videos presents two primary challenges. First, memory constraints and the "drift" inherent in autoregressive models often lead to nonsensical outputs over extended sequences. Second, traditional metrics for video quality often fail to capture whether a generated video is semantically useful for an autonomous agent's decision-making process. DriveGenVLM addresses these hurdles by combining advanced diffusion sampling techniques with VLM-based semantic validation.

Methodology

Video Generation via Flexible Diffusion

The DriveGenVLM framework utilizes Denoising Diffusion Probabilistic Models (DDPM), specifically adopting the Flexible Diffusion Model (FDM) approach. Unlike standard models that generate frames sequentially, FDM allows for the sampling of arbitrary subsets of frames conditioned on others. This flexibility is crucial for maintaining long-term temporal consistency and preventing the "forgetting" effect common in simpler autoregressive models.

Optimized Sampling Schemes

To maximize video quality and coherence, the researchers evaluated three distinct sampling strategies:

  1. Autoregressive (Autoreg): Sequential generation based on a sliding window of the previous 10 frames.
  2. Hierarchy-2: A two-tier approach that first generates "anchor" frames across the entire duration and subsequently fills the intermediate gaps.
  3. Adaptive Hierarchy-2: A dynamic strategy that selects conditioning frames to maximize visual diversity, measured by Learned Perceptual Image Patch Similarity (LPIPS) distance.

Vision Language Integration

To validate the utility of the generated sequences, the framework integrates EILEV (Efficient In-context Learning on Egocentric Videos). EILEV acts as a semantic judge, taking the generated video as input and producing natural language narrations. If the VLM can accurately describe the scene—identifying road types, lighting conditions, and vehicle behavior—it confirms that the diffusion model has captured the underlying logic of the driving environment rather than just mimicking pixel patterns.

Experimental Setup

The model was trained and evaluated using the Waymo Open Dataset. The study focused on three specific camera perspectives: Front, Front-left, and Front-right. The dataset comprised 138 videos, with 108 used for training and 30 for testing. To manage computational load while preserving essential semantic details, videos were downsampled to a resolution of 128x128. Training was conducted on an NVIDIA L4 GPU, totaling approximately 108 GPU hours across all camera angles.

Results and Discussion

Quantitative Performance

The Adaptive Hierarchy-2 sampling scheme emerged as the superior method, achieving the lowest Frechet Video Distance (FVD) scores. For instance, in the Front-left camera evaluation, Adaptive Hierarchy-2 reached an FVD of 812.425, a significant improvement over the 1401.793 recorded by the standard autoregressive approach. This suggests that global conditioning is vital for maintaining the structural integrity of generated driving scenes.

Qualitative and Semantic Success

The integration with EILEV proved highly successful. The VLM generated accurate narrations for the predicted sequences, correctly distinguishing between complex environmental factors such as "driving in a city" versus "highway driving" and identifying "daytime" versus "nighttime" conditions. This demonstrates that the diffusion model successfully encodes high-level environmental features that are recognizable to a reasoning engine.

Transfer Learning

An important finding was the efficacy of transfer learning across different viewpoints. By using weights from the front-facing camera model to initialize training for side cameras, the researchers showed that fundamental features of driving scenes are transferable, potentially reducing the data requirements for multi-camera autonomous systems.

Conclusion and Future Work

DriveGenVLM demonstrates that diffusion-based video generation can produce semantically rich data suitable for VLM-based reasoning. While the model currently faces challenges in simulating highly complex traffic interactions—such as nuanced pedestrian movements or intricate architectural details—it establishes a robust baseline for "what-if" scenario planning.

Future research will focus on integrating larger language models to move beyond simple narration toward active navigation and path planning. Bridging the gap between synthetic logic and the unpredictable nature of real-world sensor data remains a primary objective for the next generation of autonomous driving frameworks.

Summary updated

Abstract

The advancement of autonomous driving technologies necessitates increasingly sophisticated methods for understanding and predicting real-world scenarios. Vision language models (VLMs) are emerging as revolutionary tools with significant potential to influence autonomous driving. In this paper, we propose the DriveGenVLM framework to generate driving videos and use VLMs to understand them. To achieve this, we employ a video generation framework grounded in denoising diffusion probabilistic models (DDPM) aimed at predicting real-world video sequences. We then explore the adequacy of our generated videos for use in VLMs by employing a pre-trained model known as Efficient In-context Learning on Egocentric Videos (EILEV). The diffusion model is trained with the Waymo open dataset and evaluated using the Fréchet Video Distance (FVD) score to ensure the quality and realism of the generated videos. Corresponding narrations are provided by EILEV for these generated videos, which may be beneficial in the autonomous driving domain. These narrations can enhance traffic scene understanding, aid in navigation, and improve planning capabilities. The integration of video generation with VLMs in the DriveGenVLM framework represents a significant step forward in leveraging advanced AI models to address complex challenges in autonomous driving.
BibTeX
@misc{fu2024drivegenvlm,
  author = {Yongjie Fu and Anmol Jain and Xu Chen and Zhaobin Mo and Xuan Di},
  title = {{DriveGenVLM: Real-world Video Generation for Vision Language Model based Autonomous Driving}},
  year = {2024},
  eprint = {2408.16647},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2408.16647},
}
Figure