TL;DR
The paper introduces LAW (LAtent World model), a self-supervised learning approach that improves end-to-end autonomous driving by teaching models to predict future scene features based on their own planned actions. By learning to "imagine" the future in a latent feature space rather than generating raw pixels, LAW achieves state-of-the-art performance across major benchmarks like nuScenes, NAVSIM, and CARLA.
What Problem Does This Paper Address?
Traditional autonomous driving stacks are modular, passing processed data (like bounding boxes) from perception to planning. While "end-to-end" planners aim to reduce information loss by using raw sensor data directly, they face a major challenge: how to learn meaningful scene representations without an astronomical amount of manual labeling.
Existing self-supervised methods often fall short. Static techniques (like Masked Autoencoders) ignore the temporal nature of driving. Meanwhile, generative "world models" that produce future video frames are often too computationally expensive (using slow diffusion models) or require dense annotations (like occupancy grids) to be effective for real-time planning. The authors sought a way to leverage temporal data and ego-vehicle actions to learn better features without the overhead of full image generation.
Methods
The core innovation is the LAtent World model (LAW). Instead of predicting future pixels, it predicts future latent features—the high-level mathematical representations of a scene.
The process works as follows:
- Feature Extraction: A visual encoder turns raw sensor images into "visual latents."
- Action Integration: The model's predicted trajectory (its "action") is combined with these latents to create "action-aware latents."
- Future Prediction: A Transformer-based world model takes these action-aware latents and predicts what the visual latents will look like in the next time step ($T+1$).
- Self-Supervision: During training, the model is penalized (via Mean Squared Error) if its predicted future latents don't match the actual latents extracted from the real future frame.
Crucially, LAW is framework-agnostic. It can be plugged into "perception-free" models that use perspective-view features or "perception-based" models that utilize Bird's-Eye-View (BEV) representations.
Datasets
The authors evaluated LAW on three diverse benchmarks to ensure robustness:
- nuScenes: A massive real-world dataset for open-loop planning (predicting what a human driver would do).
- NAVSIM: A newer, more challenging benchmark based on the nuPlan dataset, focusing on complex driving scenarios.
- CARLA (Town05 Long): A high-fidelity simulator for closed-loop evaluation, where the model's actions actually change the environment it sees.
Key Insights and Ideas
- Action-Awareness is Essential: You cannot accurately predict the future of a driving scene without knowing how the ego-vehicle is moving. By conditioning the world model on the predicted trajectory, the features become more relevant for planning.
- Latent Space > Pixel Space: Predicting features is significantly more efficient than generating images. It allows the model to focus on the "semantics" of the scene (where are the cars? where is the road?) rather than the exact color of every pixel.
- The 1.5-Second Sweet Spot: The authors found that predicting 1.5 seconds into the future provided the best results. Predicting only 0.5 seconds ahead offered too little change to learn from, while 10 seconds was too unpredictable to provide a stable training signal.
- Transformer Superiority: Ablation studies showed that using stacked Transformer blocks for the world model outperformed simple MLPs, likely because Transformers better capture the spatial interactions between different parts of the scene.
Key Results
LAW achieved state-of-the-art (SOTA) results across the board:
- nuScenes: The perception-based version of LAW achieved an average L2 displacement error of 0.49m and a collision rate of 0.19%, outperforming established models like VAD and UniAD.
- NAVSIM: LAW reached a PDMS (Predictive Driver Model Score) of 84.6, surpassing human-imitation baselines and other SOTA planners.
- CARLA: In closed-loop testing, LAW achieved a Driving Score of 70.1, a significant jump over previous leaders like DriveAdapter (65.9) and ThinkTwice (65.0).
Limitations and Future Work
While LAW is powerful, it has a few notable constraints:
- Encoder Dependency: The "ground truth" for the world model is the output of the visual encoder. If the encoder is poorly trained or biased, the world model will learn to replicate those errors.
- Single-Step Focus: The primary experiments focused on predicting one step ahead. While the appendix explores auto-regressive multi-step prediction (e.g., predicting 1.5s then using that to predict 3.0s), this adds complexity and potential for error accumulation.
- Future Directions: The authors suggest that incorporating multi-frame history (temporal latents) could further enhance performance, as shown in their preliminary appendix experiments.
Summary updated