Abstract
End-to-end autonomous driving has garnered widespread attention. Current end-to-end approaches largely rely on the supervision from perception tasks such as detection, tracking, and map segmentation to aid in learning scene representations. However, these methods require extensive annotations, hindering the data scalability. To address this challenge, we propose a novel self-supervised method to enhance end-to-end driving without the need for costly labels. Specifically, our framework LAW uses a LAtent World model to predict future latent features based on the predicted ego actions and the latent feature of the current frame. The predicted latent features are supervised by the actually observed features in the future. This supervision jointly optimizes the latent feature learning and action prediction, which greatly enhances the driving performance. As a result, our approach achieves state-of-the-art performance in both open-loop and closed-loop benchmarks without costly annotations.
Introduction
Traditional autonomous driving stacks are modular, typically involving discrete stages for object detection, tracking, and path planning. While end-to-end (E2E) planners aim to simplify this pipeline by mapping sensor data directly to driving commands—thereby reducing information loss—they often struggle to extract high-quality, relevant features from raw sensor data.
A significant bottleneck in current E2E models is their reliance on dense manual annotations for auxiliary tasks like depth estimation or map segmentation to guide feature learning. As shown in Figure 1, these annotation-based tasks are labor-intensive and limit the ability to scale with the vast amounts of unlabeled driving data available. To overcome this, the LAtent World model (LAW) introduces a self-supervised paradigm that learns temporal, action-aware scene representations by predicting how a scene evolves in response to the vehicle's own actions.
Methodology: The LAW Framework
The core innovation of LAW is its ability to predict future latent features—abstract mathematical representations of the environment—rather than computationally expensive pixels.
Action-Aware Latents and Prediction
The framework operates by first extracting visual latents from current sensor inputs. These are then combined with the predicted ego-trajectory (waypoints) to form "action-aware" latents. This conditioning is crucial; it ensures the model understands the causal relationship between its intended path and the resulting changes in the environment.
As illustrated in Figure 2, a Transformer-based Latent World Model processes these action-aware latents to predict the scene's state at time $T+1$. During training, these predictions are compared against the actual features extracted from the subsequent frame using a Mean Squared Error (MSE) loss. This self-supervised signal forces the encoder to capture features that are most relevant for predicting future states, effectively "supervising" the representation learning without human labels.
Framework Universality
LAW is designed to be framework-agnostic. It can be integrated into perception-free models (direct image-to-trajectory) or perception-based models that utilize Bird’s-Eye-View (BEV) features and detection heads. This flexibility allows LAW to enhance a variety of existing E2E architectures.
Experimental Results
The researchers evaluated LAW across three major benchmarks, demonstrating its robustness in both open-loop and closed-loop scenarios.
- nuScenes (Open-Loop): In predicting paths from recorded logs, LAW achieved a state-of-the-art average L2 displacement error of 0.49m (perception-based), a significant improvement over the previous best of 0.72m.
- CARLA (Closed-Loop): In a high-fidelity simulator where actions affect the environment, LAW reached a Driving Score of 70.1 on the Town05 Long benchmark, outperforming established models like ThinkTwice (65.0).
- NAVSIM: On this non-reactive simulation benchmark, LAW earned a Predictive Driver Model Score (PDMS) of 84.6, surpassing UniAD (83.4).
Key Insights
Ablation studies revealed that the 1.5-second prediction horizon is optimal; shorter intervals provide insufficient learning signals, while longer ones introduce excessive uncertainty. Furthermore, the inclusion of ego-actions ("Action-aware") was found to be the "secret sauce" for effective latent prediction, as it captures the interactive nature of driving.
Conclusion
The LAW framework demonstrates that self-supervised latent prediction is a powerful alternative to annotation-heavy perception tasks for end-to-end driving. By focusing on action-conditioned future states, LAW develops richer, more predictive scene representations that lead to superior planning performance. Future work will explore extending the temporal depth of these predictions and integrating a wider variety of sensor modalities like LiDAR and Radar.
Summary updated