Driving Scene Synthesis on Free-form Trajectories with Generative Prior

Zeyu Yang, Zijie Pan, Yuankun Yang, Xiatian Zhu, Li Zhang

TL;DR

This paper introduces DriveX, a novel framework for generating realistic driving scenes along arbitrary, non-recorded trajectories. By integrating a video diffusion model as a generative prior into a 3D scene representation (3D Gaussian Splatting), DriveX overcomes limitations of existing methods that struggle to extrapolate beyond observed views, enabling more flexible and robust autonomous driving simulations.

What Problem Does This Paper Address?

Creating realistic virtual environments for training and evaluating autonomous driving systems is crucial. Current methods for synthesizing novel views of driving scenes often rely on reconstructing environments from recorded video sequences. While these methods work well for interpolating views along the exact path captured in the training data, they fail dramatically when trying to generate views from trajectories that deviate from the original recording. This limitation hinders the ability to simulate diverse driving scenarios, especially those involving unexpected maneuvers or off-road excursions, which are critical for testing the safety and robustness of autonomous systems. The core challenge is to generate consistent and high-fidelity driving scenes for novel, unseen trajectories, going beyond the constraints of the limited viewpoints available in typical driving datasets.

Methods

DriveX tackles this problem by proposing a novel framework that iteratively refines a 3D scene representation (specifically, 3D Gaussian Splatting) using a generative prior from a video diffusion model.

Here's a breakdown of the key techniques:

  • 3D Gaussian Splatting (3DGS): This serves as the underlying scene representation. 3DGS is chosen for its ability to render high-fidelity scenes in real-time and its explicit properties that are beneficial for optimization.
  • Generative Prior from Video Diffusion Model: A pre-trained video diffusion model is employed to generate "pseudo ground truth" novel view images. This model is conditioned on the current state of the 3D scene representation.
  • Inpainting-Based Video Restoration: Instead of directly asking the diffusion model to generate novel views, DriveX formulates a novel inpainting task. The diffusion model is used to restore degraded novel view renderings from the in-training 3D Gaussian model. This is guided by an "unreliability mask" that highlights regions where the 3D model is likely to be inaccurate. This formulation disentangles the generative task from identifying degradation patterns, simplifying training.
  • Iterative Refinement: The process is iterative. The 3D Gaussian model is optimized using both recorded views and the refined novel views generated by the diffusion model. This co-adaptation allows the 3D scene representation and the generative prior to mutually reinforce each other, leading to progressively better synthesis quality.
  • Novel Trajectory Sampling: To generate useful novel views, DriveX uses a "panning camera" trajectory that gradually shifts laterally from a recorded front view. This strategy balances the need for novel views with the requirement of having some reference points for the diffusion model.
  • Projected LiDAR as Auxiliary Condition: For challenging scenes, projected LiDAR point clouds are used as an additional condition to guide the generative process, further improving robustness and accuracy, especially for depth estimation.

What's new here is the tight integration of the generative prior into the optimization loop of the 3D scene representation, specifically through the inpainting-based restoration task and the iterative refinement process. This allows the generative model to learn from and adapt to the evolving 3D scene, rather than being a static, independent component.

Datasets

  • Waymo Open Dataset (WOD) [25]: This large-scale dataset, containing surrounding videos and synchronized LiDAR point clouds, is used for both training the diffusion model and evaluating the DriveX framework.
  • PandaSet dataset [35]: Used to validate the versatility of the proposed method on different scene types and sensor configurations.

The paper does not introduce a new benchmark dataset but rather focuses on evaluating performance on novel trajectories within existing datasets.

Key Insights and Ideas

  • Generative Priors as "Intelligent Guessing" for Reconstruction: The core idea is that a powerful video diffusion model, trained on vast amounts of visual data, can "guess" what a driving scene should look like from a novel viewpoint, even if the 3D model is currently poor at rendering it. This guess acts as a strong supervisory signal to improve the 3D model.
  • Inpainting Simplifies Generative Model Training: By framing the task as "inpainting" or "restoration" of degraded renderings, the diffusion model doesn't need to learn to generate entirely new scenes from scratch or explicitly handle complex degradation patterns. It focuses on refining existing, albeit imperfect, information, making training more robust and less reliant on specific degradation simulations.
  • Iterative Co-Adaptation is Key: The "aha moment" is that the 3D scene representation and the generative model can learn from each other iteratively. As the 3D model improves, it provides better inputs for the diffusion model, which in turn generates more accurate supervision, creating a virtuous cycle.
  • Unreliability Masks Guide Refinement Effectively: The unreliability mask, derived from comparing rendered and warped images, is crucial for focusing the generative model's efforts on the most problematic areas of the scene, preventing it from "hallucinating" details in reliable regions.

Key Results

  • Significant Improvement on Novel Trajectories: DriveX demonstrates substantial improvements in view synthesis quality for novel trajectories, particularly when compared to state-of-the-art reconstruction-only methods. For a 3m lateral shift, DriveX achieved a 14.3% decrease in FID and a 56.9% increase in IoU compared to StreetGaussian [37].
  • High-Fidelity and Consistent Synthesis: Qualitative results show that DriveX can generate realistic and consistent views even at significant deviations from recorded trajectories, preserving safety-critical elements like lanes and vehicles.
  • Real-Time Inference Speed: Once the 3D Gaussian model is trained, inference is fast, leveraging the efficiency of 3DGS.
  • Robustness to Aggressive View Changes: The method shows resilience to challenging scenarios like large lateral shifts and significant pitch angles, outperforming baselines that fail to synthesize meaningful images.
  • Effective Application on AI-Generated Videos: DriveX can also be used to enhance the realism of AI-generated driving videos, demonstrating its versatility.

Limitations and Future Work

  • Reconstruction Time: The paper acknowledges that incorporating a diffusion-based generative model significantly increases reconstruction time. However, they suggest that a more lightweight model might suffice in the future, given the inpainting formulation.
  • Fixed Trajectory Sampling: The current method uses a simple panning camera scheme for sampling novel trajectories. Future work could explore adaptive strategies to explore optimal novel trajectories during optimization.
  • Unseen Regions: The framework struggles to reconstruct feasible appearances from scratch for regions of the scene that are entirely invisible from all recorded views, lacking initial geometry initialization.
  • Future Work: The authors suggest exploring more lightweight generative models and adaptive trajectory sampling strategies. They also highlight the potential for DriveX to be applied to AI-generated videos, which presents further challenges and opportunities.

Summary updated

Abstract

Driving scene synthesis along free-form trajectories is essential for driving simulations to enable closed-loop evaluation of end-to-end driving policies. While existing methods excel at novel view synthesis on recorded trajectories, they face challenges with novel trajectories due to limited views of driving videos and the vastness of driving environments. To tackle this challenge, we propose a novel free-form driving view synthesis approach, dubbed DriveX, by leveraging video generative prior to optimize a 3D model across a variety of trajectories. Concretely, we crafted an inverse problem that enables a video diffusion model to be utilized as a prior for many-trajectory optimization of a parametric 3D model (e.g., Gaussian splatting). To seamlessly use the generative prior, we iteratively conduct this process during optimization. Our resulting model can produce high-fidelity virtual driving environments outside the recorded trajectory, enabling free-form trajectory driving simulation. Beyond real driving scenes, DriveX can also be utilized to simulate virtual driving worlds from AI-generated videos.
BibTeX
@inproceedings{yang2025driving,
  author = {Zeyu Yang and Zijie Pan and Yuankun Yang and Xiatian Zhu and Li Zhang},
  title = {{Driving Scene Synthesis on Free-form Trajectories with Generative Prior}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2025},
}
Figure