Abstract
World model-based reinforcement learning (RL) has emerged as a promising approach for autonomous driving, utilizing a learned latent dynamics model to train planning policies. To accelerate learning, researchers often employ a pretrain-finetune paradigm, where online RL is initialized with a model and policy learned from offline data. However, transitioning to new tasks often triggers significant performance degradation due to distribution shifts. We identify two primary causes for this "performance dip": mismatches in the planning policy and the dynamics model. To address this, we introduce AdaWM (Adaptive World Model), a planning method featuring mismatch identification and alignment-driven finetuning. By selectively updating components using efficient low-rank updates, AdaWM ensures robust and efficient adaptation. Extensive experiments in the CARLA simulator demonstrate that AdaWM significantly outperforms existing baselines in complex driving scenarios.
Introduction
In the quest for safe and efficient autonomous driving, the ability to adapt to new environments—such as different weather conditions, traffic densities, or road layouts—is critical. While training from scratch is computationally expensive and potentially dangerous, the standard alternative—finetuning a pretrained model—often suffers from a "catastrophic" performance drop during the initial stages of online interaction. This phenomenon occurs because the agent's internal representation of the world (the world model) and its decision-making strategy (the policy) are no longer aligned with the new environment's dynamics.
The Challenge of Distribution Shift
The core difficulty in the pretrain-finetune paradigm is distribution shift. When an agent trained on right-hand turns is suddenly tasked with navigating a dense roundabout, it faces two distinct types of errors:
- Dynamics Model Mismatch: The agent's internal "physics engine" fails to predict how the environment will evolve, leading to inaccurate simulations of future states.
- Policy Mismatch: Even if the model were perfect, the agent's learned maneuvers may be inappropriate for the new task's specific requirements.
Naive finetuning strategies typically update all parameters simultaneously. However, this often leads to noisy updates that overwrite useful pretrained knowledge, resulting in the observed performance crash.
The AdaWM Framework
AdaWM introduces a surgical approach to finetuning through a two-step process: Mismatch Identification and Alignment-driven Finetuning.
1. Mismatch Identification
Rather than assuming both the model and policy are equally flawed, AdaWM quantifies the specific source of error at each step. It utilizes Total Variation (TV) distance to measure the shift in both the dynamics model and the policy. This allows the system to determine whether the performance bottleneck is a misunderstanding of the world's "physics" or a failure in decision-making logic.
2. Alignment-driven Finetuning
Once the dominant mismatch is identified, AdaWM selectively updates the corresponding component. This prevents the policy from trying to learn based on a broken world model, or vice versa. To maintain efficiency and stability, the method employs:
- Low-Rank Adaptation (LoRA): For the World Model, AdaWM updates only a small set of low-dimensional vectors, preserving the core knowledge of the massive neural network while allowing for rapid adaptation.
- Weighted Convex Ensembles: The policy is treated as a collection of sub-units, where finetuning focuses on updating the weights of these units to streamline the learning process.
Experimental Evaluation
We evaluated AdaWM using the CARLA simulator, a high-fidelity 3D environment. The testing suite included the Bench2Drive benchmark, encompassing 2 million frames across 44 scenarios and 23 weather conditions. We specifically focused on challenging transitions, such as moving from simple right turns in light traffic to dense-traffic roundabouts (ROM03) and complex left turns (LTD03).
Key Results
AdaWM demonstrated substantial improvements over state-of-the-art baselines, including DreamerV3, VAD, and UniAD:
- Success Rate (SR): In the highly demanding LTD03 task (Left Turn, Dense Traffic), AdaWM achieved a 70% success rate, doubling the performance of the strongest baseline, DreamerV3 (35%). Supervised methods like UniAD and VAD struggled to exceed a 10% success rate in these interactive settings.
- Safety Metrics: Safety is quantified by Time-to-Collision (TTC). In the ROM03 task, AdaWM maintained a TTC of 2.05, significantly higher than the baseline average of approximately 0.95, indicating much safer driving behavior.
- Stability: By identifying the "Model-First" priority—fixing the world model before the policy—AdaWM successfully mitigated the performance crash typically seen at the start of online finetuning.
Discussion and Future Work
AdaWM proves that selective, alignment-driven updates are more effective than global finetuning for autonomous agents. However, several avenues for future research remain. Currently, the method relies on a threshold parameter ($C$) to switch between model and policy updates; automating the selection of this parameter would improve generalizability. Additionally, while AdaWM handles dense traffic well, explicitly modeling the social negotiations inherent in multi-agent interactions could further enhance performance in complex urban environments.
By addressing the root causes of performance degradation during finetuning, AdaWM provides a robust framework for deploying autonomous driving systems that can safely and efficiently adapt to the ever-changing real world.
Summary updated