PosePilot: Steering Camera Pose for Generative World Models with Self-supervised Depth

Bu Jin, Weize Li, Baihan Yang, Zhenxin Zhu, Junpeng Jiang, Huan-ang Gao, Haiyang Sun, Kun Zhan, Hengtong Hu, Xueyang Zhang, Peng Jia, Hao Zhao

Abstract

Generative world models for autonomous driving often lack precise camera pose control, leading to geometric inconsistencies such as viewpoint drift. PosePilot introduces a lightweight, architecture-agnostic framework that grounds video generation in 3D geometry using self-supervised depth estimation and structure-from-motion (SfM) principles. By incorporating depth and pose readouts alongside photometric control losses, PosePilot ensures that generated sequences remain consistent with specified trajectories. Experimental results on nuScenes and RealEstate10K demonstrate that PosePilot significantly reduces translation and rotation errors across various generative architectures while improving overall visual realism.

Introduction

Generative world models represent the frontier of autonomous driving (AD), enabling AI systems to "imagine" future scenarios and navigate rare edge cases. However, a persistent challenge is pose controllability. Existing models often rely on black-box attention layers to learn pose features, which frequently results in "drifting" viewpoints that fail to accurately follow requested steering or movement commands. PosePilot bridges this gap by grounding video generation in the physical laws of optics and geometry, ensuring that the generated world is not just visually plausible but geometrically sound.

Methodology

The core innovation of PosePilot is the shift from purely learned pose features to explicit geometric supervision. The framework is architecture-agnostic, allowing it to be "plugged" into existing diffusion-based models (e.g., Vista, DiVE) or autoregressive models (e.g., DrivingWorld) without altering their internal structures. The module adds approximately 180 million parameters, a relatively small footprint compared to the billion-parameter models it supervises.

PosePilot Framework Architecture

The framework consists of several key components:

  • Depth and Pose Readouts: Lightweight modules extract depth maps and relative camera motion directly from generated video sequences, creating a feedback loop.
  • Pose-aware Forward Warping: Based on SfM principles, the model projects pixels from one frame to the next using estimated depth and pose.
  • Photometric Control Loss: A loss function compares warped frames to generated frames, penalizing discrepancies to force alignment with requested 3D movement.
  • Bidirectional Consistency: Inverse warping (backward in time) handles occlusions and ensures scene stability from both directions.
  • Pose Regression: Direct Mean Squared Error (MSE) loss is applied to pose readouts to minimize drift during complex maneuvers.

Evaluation and Results

PosePilot was evaluated on the nuScenes dataset for urban driving and RealEstate10K for general scene walkthroughs. The results indicate that geometry acts as a powerful prior, making the learning process more efficient and the output more precise.

Quantitative Performance Comparison on nuScenes

Key findings include:

  • Precision Improvements: On the nuScenes dataset, adding PosePilot to the DiVE model reduced Translation Error (TranErr) by nearly 50% (from 13.07 to 6.37) and significantly lowered Rotation Error (RotErr).
  • Enhanced Visual Quality: Improvements in Fréchet Inception Distance (FID) suggest that better geometric grounding leads to fewer visual artifacts and flickers, resulting in higher realism.
  • Cross-Domain Generalization: PosePilot outperformed specialized pose-control models like CameraCtrl on RealEstate10K, proving that its geometric approach is universal and not limited to driving scenarios.

Conclusion and Future Work

PosePilot demonstrates that integrating 3D geometric constraints into generative models significantly enhances pose control and video realism. While highly effective, current limitations include the "static world" assumption in warping logic, which can be challenged by fast-moving dynamic objects. Additionally, monocular depth estimation inherently struggles with absolute scale. Future work may involve integrating LiDAR or radar data to ground the "imagined" world in real-world meters and improving robustness under extreme lighting conditions.

Reference: Jin, B., et al. (2025). PosePilot: Steering Camera Pose for Generative World Models with Self-supervised Depth. arXiv:2505.01729v2.

Summary updated

Abstract

Recent advancements in autonomous driving (AD) systems have highlighted the potential of world models in achieving robust and generalizable performance across both ordinary and challenging driving conditions. However, a key challenge remains: precise and flexible camera pose control, which is crucial for accurate viewpoint transformation and realistic simulation of scene dynamics. In this paper, we introduce PosePilot, a lightweight yet powerful framework that significantly enhances camera pose controllability in generative world models. Drawing inspiration from self-supervised depth estimation, PosePilot leverages structure-from-motion principles to establish a tight coupling between camera pose and video generation. Specifically, we incorporate self-supervised depth and pose readouts, allowing the model to infer depth and relative camera motion directly from video sequences. These outputs drive pose-aware frame warping, guided by a photometric warping loss that enforces geometric consistency across synthesized frames. To further refine camera pose estimation, we introduce a reverse warping step and a pose regression loss, improving viewpoint precision and adaptability. Extensive experiments on autonomous driving and general-domain video datasets demonstrate that PosePilot significantly enhances structural understanding and motion reasoning in both diffusion-based and auto-regressive world models. By steering camera pose with self-supervised depth, PosePilot sets a new benchmark for pose controllability, enabling physically consistent, reliable viewpoint synthesis in generative world models.
BibTeX
@inproceedings{jin2025posepilot,
  author = {Bu Jin and Weize Li and Baihan Yang and Zhenxin Zhu and Junpeng Jiang and Huan-ang Gao and Haiyang Sun and Kun Zhan and Hengtong Hu and Xueyang Zhang and Peng Jia and Hao Zhao},
  title = {{PosePilot: Steering Camera Pose for Generative World Models with Self-supervised Depth}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2025},
  url = {https://openreview.net/forum?id=gJlq519bvU},
}
Figure