SAFER: Navigating the Exploration-Exploitation Dilemma in World-Model-Based Planning for Autonomous Driving

Yansong Qu, Zilin Huang, Zihao Sheng, Sikai Chen, Samuel LabiarXiv 2026 · arXiv

TL;DR

SAFER is a world-model-based planning framework for autonomous driving that solves the exploration-exploitation dilemma by learning from "staged" failures—mistakes made by weaker agents but corrected by an expert. By combining offline learning from failures with a fast, risk-aware latent world model and online refinement, SAFER achieves state-of-the-art safety and route completion in CARLA and MetaDrive.

What Problem Does This Paper Address?

Autonomous vehicles (AVs) face a fundamental exploration-exploitation dilemma. To drive safely, an AV needs a "world model"—a mental simulator that predicts the consequences of its actions. However, a world model is only as good as its training data.

Most driving datasets consist of "competent driving" (logs of humans or experts driving well). If a planner only exploits this data, it can't handle rare, critical edge cases because it has never seen them. If it tries to explore beyond its data, it hits the "optimizer's curse": the planner is naturally drawn to actions where the model's errors lead to overestimated rewards, often resulting in catastrophic failures.

Standard Reinforcement Learning (RL) solves this by letting the agent fail and learn from its own crashes. In the real world, this is unacceptable. SAFER addresses this by asking: How can we broaden a model's coverage of failures without the agent having to crash itself on the road?

Methods

The authors propose the SAFER framework, an acronym for its five stages:

  1. Stage & Annotate (Offline): Instead of just using expert logs, the authors deliberately "stage" failures in simulation. They use partially trained (weaker) agents and scripted perturbations (like withholding brakes) to create dangerous situations. Crucially, an expert (a privileged PPO agent) annotates every state with the correct action and value. This teaches the model not just what a crash looks like, but how to recover.
  2. Foresee: SAFER uses a chunked latent world model. Unlike traditional models that predict the future step-by-step (which is slow), this model predicts a whole "chunk" of the future (1.6 seconds) in a single transformer pass. This makes real-time planning feasible.
  3. Evaluate: A risk-aware Model Predictive Path Integral (MPPI) planner samples candidate maneuvers. It scores them based on predicted rewards minus a heavy penalty for collision risk.
  4. Innovation: They introduce braking monotonicity. Usually, models might correlate "hard braking" with "danger" because they co-occur in data. SAFER uses a soft constraint to ensure that braking harder never increases the predicted risk, preventing the planner from being afraid to stop.
  5. Refine (Online): As the vehicle drives, it continuously fine-tunes its dynamics, reward, and risk models on its own experience. To keep the planning objective stable, the planner scores candidates using Exponential Moving Average (EMA) copies of these heads (the "target network" idea from value learning).

Datasets

The framework was evaluated in two high-fidelity simulators:

  • CARLA: Used for closed-loop evaluation on 76 routes across various towns (Town01–Town06), including held-out test environments.
  • MetaDrive: Evaluated on the CAT (Closed-loop Adversarial Training) benchmark, which uses 500 scenarios from the Waymo Open Motion Dataset. This includes "safety-critical" traffic where an adversarial generator tries to force the ego vehicle into a collision.

Key Insights and Ideas

  • The Parenting Analogy: The authors liken SAFER to how children learn. A child doesn't have to touch every hot stove to learn about burns; parents point to negative examples and explain the consequences. SAFER’s offline staged failures are those "parental lessons."
  • Learning from "Others' Failures": By using weaker agents to generate the failures and an expert to label the "correct" path, the model learns the boundaries of the state space safely.
  • Chunking for Speed: The move from autoregressive (step-by-step) prediction to chunked prediction is vital for robotics. It allows the planner to evaluate 135 different future trajectories at 10Hz on a single GPU.
  • Risk as a Cumulative Target: Instead of just predicting if a collision happens at step $k$, the risk head predicts if a collision has happened by step $k$. This cumulative approach naturally weights early collisions more heavily.

Key Results

SAFER outperformed several strong baselines, including DreamerV3, TD-MPC2, and various imitation learning methods.

  • CARLA: Reached a route completion of 0.95 and a reward per step of 0.77. It significantly reduced pedestrian collisions compared to the behavior-cloned prior (from 19.5 to 8.8 per 100km).
  • MetaDrive: Achieved the highest episode reward (53.4 in regular traffic, 49.0 in safety-critical traffic). In safety-critical scenarios, it maintained an average speed of 10 m/s, while the next best model (CurricuVLM) dropped to 9.45 m/s.
  • Efficiency: The entire model is relatively small (15.5M parameters) and runs at 43–45 Hz on a single Quadro RTX 6000 GPU, proving that world-model planning doesn't require massive compute clusters.

Limitations and Future Work

  • Privileged Perception: The current implementation relies on a privileged Bird’s-Eye View (BEV) input (a semantic map of the surroundings). While this isolates the planning logic, real-world deployment would require integrating this with raw sensor data (cameras/LiDAR) and handling perception noise.
  • Expert Dependency: The "Annotate" stage requires a high-quality expert that can be queried at any state. In complex real-world scenarios, defining or training such an expert might be difficult.
  • Static Expert: The online refinement stage updates the world model but keeps the expert-provided "prior" and "value" heads frozen. Future work could explore ways to improve the expert labels themselves through online interaction.

Summary updated

Abstract

An autonomous vehicle must foresee the consequences of its actions, especially in the rare, time-critical situations where crashes happen. A world model offers this foresight, but only within the coverage of its training data, which poses an exploration-exploitation dilemma. Within this coverage a planner can hardly improve on the data, and beyond it the model's errors mislead it. Logs of competent driving rarely contain the failures that would widen the coverage, and agents that learn from interaction gain them only by failing themselves, without being shown what to do instead. We propose SAFER, a framework that learns offline from failures that others make, judged by an expert, and keeps learning online from its own driving. Offline, partially trained agents and scripted, recoverable perturbations Stage failures in simulation, and an expert Annotates every visited state with its action and value, so that much of what the planner explores is covered before it drives. In planning, a chunked latent world model Foresees the consequences fast enough for real-time use, and a risk-aware sampling-based planner Evaluates candidate maneuvers by their return minus a bounded collision penalty. Online, a loop continuously Refines the model wherever this driving leaves the coverage of the offline data. In closed-loop evaluation against the compared methods, SAFER reaches the highest route completion and reward per step on the CARLA Leaderboard routes, 0.95 and 0.77 from a privileged bird's-eye view, and the highest episode reward in MetaDrive, 53.4 in regular and 49.0 in safety-critical traffic. Project page: https://ys-qu.github.io/safer-website.
BibTeX
@misc{qu2026safer,
  author = {Yansong Qu and Zilin Huang and Zihao Sheng and Sikai Chen and Samuel Labi},
  title = {{SAFER: Navigating the Exploration-Exploitation Dilemma in World-Model-Based Planning for Autonomous Driving}},
  year = {2026},
  eprint = {2611.20107},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.20107},
}
Figure