Driver-WM: A Driver-Centric Traffic-Conditioned Latent World Model for In-Cabin Dynamics Rollout

Haozhuang Chi, Daosheng Qiu, Hao Su, Haochen Liu, Zirui Li, Haoruo Zhang, Chen LvarXiv 2026 · arXiv

TL;DR

Driver-WM introduces a driver-centric, traffic-conditioned latent world model that predicts future in-cabin driver dynamics by causally conditioning them on external traffic context. By shifting driver monitoring from reactive, frame-by-frame recognition to proactive, long-horizon forecasting, it provides a crucial safety layer for shared-control transitions in L2/L3 autonomous driving.

What Problem Does This Paper Address?

In SAE Level 2 and 3 autonomous driving, control frequently shifts between the automated system and the human driver. Safely managing these transitions requires anticipating whether a driver is ready to take over before a critical situation occurs.

However, a major gap exists between how AI models handle the outside world versus the cabin:

  1. Environment-centric world models excel at forecasting how roads and surrounding traffic will evolve, but they treat the human driver as an unmodeled, external source of risk.
  2. Driver Monitoring Systems (DMS) focus strictly on instantaneous, reactive classification (e.g., detecting if a driver is currently distracted or drowsy) rather than predicting how their physical and cognitive state will evolve in response to upcoming traffic events.
Fig. 1: The comparison of three paradigms: (a) Regular driver monitoring systems (DMS) for driver-state recognition. (b) Standard world models for future environment forecasting. (c) Driver-WM (ours) that performs multi-step rollout of internal driver dynamics explicitly conditioned on synchronized external traffic observations.

Driver-WM bridges this gap by treating the driver as an active dynamical system tightly coupled with the external driving environment.

Methods

Driver-WM avoids the high computational overhead and hallucination risks of pixel-level video generation by operating entirely within a compact latent space.

Fig. 1: The comparison of three paradigms: (a) Regular driver monitoring systems (DMS) for driver-state recognition. (b) Standard world models for future environment forecasting. (c) Driver-WM (ours) that performs multi-step rollout of internal driver dynamics explicitly conditioned on synchronized external traffic observations.

Key components of the architecture include:

  • Frozen VLM Interface: The model uses a frozen Vision-Language Model (Qwen3-VL) as a perceptual encoder. To ensure the features capture interaction-aware semantics rather than generic object recognition, every frame is processed alongside a fixed text prompt directing the model to focus on traffic cues and driver responses.
  • Dual-Stream Latent Architecture: It separately encodes external traffic context (from multi-view out-cabin cameras) and internal driver states (from the in-cabin camera).
  • Directed Gated Causal Injection: To model how the environment shapes human reactions, the external and internal streams are directionally coupled. A learned vector gate ($g_t$) dynamically modulates how much external context perturbs the internal driver state transition, while strictly enforcing temporal causality (ensuring the model never looks ahead into the future traffic suffix).
  • Structured Kinematic Decoding: The rolled-out internal latents are decoded into continuous 2D skeleton trajectories (136 HALPE keypoints, including detailed hand and face tracking) using a Spatial-Temporal Graph Convolutional Network (ST-GCN).
  • Auxiliary Semantic Regularization: Lightweight heads simultaneously predict categorical labels (driver behavior, emotion, traffic context, and vehicle condition) to regularize and structure the latent space during training.

Datasets

The authors evaluated Driver-WM on the AIDE assistive driving benchmark, utilizing its official split of 1,884 clips for training, 405 for validation, and 609 for testing.

Because naturalistic driving datasets are heavily dominated by low-motion, passive driving, the authors introduced a High-Motion (HM) subset (the top 10% of test clips ranked by ground-truth frame-to-frame keypoint displacement). This subset isolates safety-critical, reactive moments—such as sudden steering or braking—to rigorously test the model's predictive capabilities.

Key Insights and Ideas

  • The Mean-Regression Trap: Under standard $\ell_2$ metrics, a naive "Zero-Velocity" baseline (which simply copies the last observed frame into the future) deceptively outperforms advanced models on average because most driving clips feature very little movement. However, this baseline fails catastrophically on the safety-critical High-Motion tail. Driver-WM successfully circumvents this trap to capture true reactive dynamics.
  • Skeletons over Pixels: Choosing structured skeleton keypoints over raw video generation ensures that the model enforces physical and geometric constraints (like constant bone lengths and temporal smoothness) while remaining highly efficient.
  • Context-Driven Transitions: Driver motion inside a cabin is not isolated; it is triggered by external events. The paper proves that explicitly parameterizing the directional coupling from traffic to the driver is essential for long-horizon accuracy.

Key Results

  • Long-Horizon Fidelity: On the safety-critical High-Motion tail, Driver-WM achieves a horizon-averaged Mean Per-Joint Position Error (MPJPE) of 138.03 pixels, significantly outperforming specialized motion-forecasting baselines like MotionBERT (141.53 px) and avoiding the rapid error accumulation seen in simpler models over longer horizons.
  • Semantic Alignment: Driver-WM achieves a 90.15 Macro-F1 score for Traffic Context Recognition and 68.07 for Driver Behavior Recognition, demonstrating excellent alignment between physical rollouts and semantic understanding.
  • Sensitivity to Interventions: Through test-time causal interventions (e.g., completely removing external features or forcing unmodulated context injection), the authors demonstrated that the model actively relies on the external-to-internal pathway. Disabling the injection pathway caused the largest factual-rollout deviations, particularly on hand joints, confirming that the model successfully simulates traffic-reactive behavior.

Limitations and Future Work

  • 2D vs. 3D Kinematics: The current model forecasts 2D image-plane skeletons. Extending this framework to metric 3D space via monocular depth lifting or multi-view cabin sensors would allow cleaner integration with vehicle-frame planning modules.
  • Closed-Loop Evaluation: While Driver-WM serves as an effective risk-ranking proxy, future work should evaluate it within a closed-loop human-in-the-loop simulation to see how autonomous planning policies adapt to its short-horizon driver-response rollouts.

Summary updated

Abstract

Safe L2/L3 driving automation requires anticipating human-in-the-loop reactions during shared-control transitions. While most driving world models forecast the external environment, in-cabin intelligence remains strictly recognition-oriented and lacks multi-step rollout capabilities for driver dynamics. We introduce Driver-WM, a driver-centric latent world model that rolls out in-cabin dynamics causally conditioned on out-cabin traffic context. This formulation unifies physical kinematics forecasting with auxiliary behavioral and emotional semantic recognition. Operating in a compact latent space constructed from frozen vision-language features, Driver-WM adopts a dual-stream architecture to separately encode external traffic and internal driver states. These streams are directionally coupled via a gated causal injection mechanism, which uses a learned vector gate to modulate external contextual perturbations while strictly enforcing temporal causality. Experiments on AIDE show robust long-horizon forecasting on reactive high-motion clips, improved driver/traffic semantic alignment, and controlled interventions that expose the external-to-internal mechanism.
BibTeX
@misc{chi2026driver,
  author = {Haozhuang Chi and Daosheng Qiu and Hao Su and Haochen Liu and Zirui Li and Haoruo Zhang and Chen Lv},
  title = {{Driver-WM: A Driver-Centric Traffic-Conditioned Latent World Model for In-Cabin Dynamics Rollout}},
  year = {2026},
  eprint = {2605.05092},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2605.05092},
}
Figure