BirdDreams: Joint Perspective and BEV Video Generation for a Plausible, Reactive Driving World Model

Junyoung Seo, Jaewoo Seo, Seungryong Kim, Laura Leal-Taixé, Xiuming ZhangarXiv 2026 · arXiv

TL;DR

BirdDreams is a novel driving world model that jointly generates photorealistic perspective video and schematic bird’s-eye-view (BEV) video. By using the BEV as a "geometric interface," the model achieves unprecedented precision in ego-trajectory control and generates realistic, reactive behaviors from surrounding traffic without requiring HD maps or pre-defined vehicle trajectories at inference.

What Problem Does This Paper Address?

Simulating the "unfolding" of a driving scene is a cornerstone of autonomous vehicle (AV) development. However, building a world model that is both visually realistic and physically controllable is notoriously difficult.

Most current models struggle with two main issues:

  1. Entangled Motion: In standard perspective (RGB) video, a vehicle's motion is entangled with its appearance, lighting, and camera perspective. Occlusions make it hard for models to track "hidden" dynamics.
  2. Lack of Reactivity: Many models condition their generation on fixed "layouts" (where other cars are supposed to go). This means if you change the ego-vehicle's path, the surrounding cars don't react—they just blindly follow their pre-recorded logs.

BirdDreams tackles these by making the scene layout part of the predicted future, rather than an external input, and by leveraging the metric consistency of a top-down view.

Methods

The core of BirdDreams is a Dual-Stream Causal Diffusion Transformer (DiT).

The BirdDreams architecture uses joint attention to couple perspective and BEV streams.

Key technical innovations include:

  • Joint Perspective-BEV Generation: The model features two parallel branches—one for RGB and one for schematic BEV. "Joint attention" layers allow these streams to talk to each other. The RGB stream learns about traffic lights and textures, while the BEV stream provides a metric, occlusion-free map of vehicle positions.
  • BEV-Aware Action Conditioning: To control the ego-vehicle, the authors don't just feed in coordinates. They take the last predicted BEV frame and mathematically reproject (rotate and translate) it along the desired trajectory. This "warped" view acts as a soft hint, showing the model how the static world should shift if the car moves as intended.
  • Per-Stream Noise Sampling: During training, the model is exposed to chunks where one stream might be "cleaner" than the other. This forces the RGB branch to learn to "read" the BEV map to place cars correctly, and vice versa.
  • Autoregressive Rollout: The model generates video in chunks, caching history to maintain temporal consistency over long horizons.

Datasets

The model was trained and evaluated on:

  • RDS-HQ-1M: A massive internal dataset (~670 hours) used for pre-training.
  • nuScenes: A standard public benchmark for AV research.
  • PhysicalAI-AV: A subset of the PhysicalAI-Autonomous-Vehicles dataset released by NVIDIA.

Key Insights and Ideas

  • The BEV as a Geometric Anchor: The "aha moment" here is realizing that perspective video is a poor interface for metric control. By forcing the model to generate a BEV video at a fixed metric scale (e.g., 1 pixel = 0.14 meters), the laws of physics and geometry become much easier for the transformer to learn.
  • Reactivity is Emergent: Because the model predicts the BEV layout chunk-by-chunk, it can simulate interactions. If the ego-vehicle cuts off a generated car in the BEV space, the model learns that the "plausible" next frame involves that car slowing down to avoid a collision.
  • Flexible Conditioning: Because of the dual-stream nature, the model is a "Swiss Army knife." You can start with an image and get a BEV, or start with a hand-drawn BEV layout and generate a photorealistic video.
BirdDreams can generate videos from various starting points, including just an RGB frame or just a BEV layout.

Key Results

BirdDreams significantly outperforms prior state-of-the-art models like Vista, UniFuture, and the original OmniDreams:

  • Trajectory Fidelity: When given a specific path to follow, BirdDreams reduced the Average Displacement Error (ADE) by 35-36% compared to baselines.
  • Visual Quality: It achieved lower (better) Fréchet Video Distance (FVD) and FID scores, indicating more realistic video synthesis.
  • Human Preference: In a pairwise study, humans preferred BirdDreams' reactive behavior 57% of the time, compared to only 24% for the baseline. The baseline was frequently flagged for "impossible motion" (cars teleporting or sliding) which BirdDreams largely avoided.
Qualitative comparison showing BirdDreams maintaining spatial coherence and realistic maneuvers where baselines fail.

Limitations and Future Work

  • Schematic Simplicity: The BEV is currently schematic (simple shapes). While this helps focus on dynamics, it might miss subtle 3D interactions like height variations or complex curb geometries.
  • Inference Speed: As an autoregressive diffusion model, generating long high-resolution videos is computationally expensive.
  • Action Prediction: While the paper explores a "World-Action Model" extension, the authors note that fully integrating policy learning (deciding how to drive) with the world model is a major future direction.

Summary updated

Abstract

Driving world models aim to simulate how a scene unfolds under different ego actions. This requires plausible motion of surrounding vehicles as well as precise ego-trajectory control. Learning vehicle motion from perspective video alone is challenging because motion is entangled with appearance and vehicles can be occluded. Prior work therefore conditions video generation on layouts that specify vehicle motion, providing control over surrounding vehicles but requiring their future trajectories as inputs. We introduce BirdDreams, an autoregressive driving world model that jointly generates perspective video and a schematic bird's-eye-view (BEV) video, making scene layout part of the predicted future. The BEV, rendered from scene annotations for training, depicts roads and vehicles as simple shapes at a fixed metric scale, providing a top-down view of their positions and motion with reduced occlusion between vehicles. Perspective video, by contrast, retains scene details that the BEV omits, such as traffic signals and object appearance. Joint attention between two video diffusion transformers, one per stream, lets each stream inform the other during generation. The predicted BEV also provides a geometric interface for ego control: we translate and rotate its latest frame along an input ego trajectory and use the resulting frames as a soft condition that shows how the scene would appear to move if only the ego vehicle moved. At inference, BirdDreams generates both videos from a single camera image without requiring an HD map or vehicle-state inputs. On PhysicalAI-AV and nuScenes, BirdDreams follows input ego trajectories more faithfully than prior driving world models and generates more plausible ego motion without ego-action conditioning. Human evaluation shows that BirdDreams outperforms a perspective-only baseline in generating realistic surrounding-vehicle behavior.
BibTeX
@misc{seo2026birddreams,
  author = {Junyoung Seo and Jaewoo Seo and Seungryong Kim and Laura Leal-Taixé and Xiuming Zhang and Seung Wook Kim},
  title = {{BirdDreams: Joint Perspective and BEV Video Generation for a Plausible, Reactive Driving World Model}},
  year = {2026},
  eprint = {2611.14815},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.14815},
}
Figure