World Models for Predictive Safety in Reinforcement Learning

This direction is merged into another direction: safe exploration. Its last version stays readable and citable here.

Version 1 of 117 papers23 conceptsfrozen 2026-10-04

In traditional reinforcement learning, agents learn through trial and error. While this works for video games, it is often unacceptable in the real world. A robot learning to walk cannot afford to "try" falling down a flight of stairs, and a network controller cannot "try" dropping thousands of calls to see what happens.

To bridge this gap, researchers are moving toward predictive safety mechanisms powered by world models. Instead of learning from actual failures, these agents "imagine" the consequences of their actions in a learned simulator. By forecasting the near future, they can identify and discard dangerous trajectories before they ever occur in reality.

The World Model as a Mental Sandbox

The core intuition behind this research is that an agent should possess a "mental model" of its environment—a world model. This model allows the agent to perform a "look-ahead" search.

As Yixia Li, Heng Ji, and their colleagues demonstrate in From Word to World: Can Large Language Models be Implicit Text-based World Models?, even Large Language Models can be fine-tuned to act as these simulators. In their framework, the world model acts as a safety valve: if the model predicts that an action leads to an irreversible negative state, the agent can "rewind" its decision-making process and explore a safer alternative. This effectively turns a high-stakes real-world environment into a low-stakes sandbox.

Anticipating Failure with Termination Heads

How does an agent actually "see" a crash coming? One effective mechanism is the integration of an explicit termination prediction head within the world model.

In Hierarchical World Models as Visual Whole-Body Humanoid Controllers, Nicklas Hansen and Yann LeCun use this to control complex humanoid robots. By training the model to specifically estimate the probability of an episode ending (e.g., the robot falling), the agent can evaluate candidate trajectories and prioritize those that maximize stability. This allows the robot to navigate treacherous terrain like stairs or gaps by proactively avoiding paths with a high predicted "termination" probability.

Shielding and Safe Baselines

In industrial applications, safety isn't just about avoiding a "game over" screen; it's about maintaining a minimum level of performance. Saman Feghhi and Erik Aumayr explore this in the context of cellular networks in Safe Reinforcement Learning for Antenna Tilt Optimisation using Shielding and Multiple Baselines.

They employ a "shielding" mechanism:

  1. The agent proposes an action.
  2. The world model forecasts the outcome.
  3. The action is only executed if the predicted performance exceeds a known safe baseline.

This "anytime-competitive" approach, also championed by Jianyi Yang and Adam Wierman in Anytime-Competitive Reinforcement Learning with Policy Prior, ensures that the agent never performs worse than a conservative, pre-existing controller. Model-based planning is the "look-ahead" engine that makes these guarantees possible, preventing the agent from drifting into "dead-end" states where no safe actions remain.

Managing Uncertainty with Ensembles

A world model is only useful if it is accurate. If the model is overconfident about a dangerous path, the agent will crash. To solve this, researchers use model ensembles to quantify uncertainty.

In Safe Reinforcement Learning by Imagining the Near Future, Tengyu Ma and Garrett Thomas describe an algorithm (SMBPO) that uses multiple models to "imagine" the future. If the models disagree or if the ensemble predicts a high likelihood of failure, the agent applies a heavy reward penalty. Similarly, the CARL framework by Jesse Zhang and Sergey Levine in Cautious Adaptation For Reinforcement Learning in Safety-Critical Settings uses these learned safety models to discard risky trajectories during the transition from simulation to the real world.

Learning from Human Intervention

Finally, safety can be learned directly from human behavior. In the LaND framework, Gregory Kahn and Pieter Abbeel show in LaND: Learning to Navigate From Disengagements that human interventions (disengagements) can serve as a primary learning signal.

By training a model to predict when a human is likely to take over, the robot learns a "safety-aware cost function." During deployment, the robot generates many possible paths and simply picks the one with the lowest predicted probability of requiring human intervention. This turns the world model into a proxy for human judgment, allowing for autonomous navigation in complex environments like busy sidewalks.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_world_models_for_predictive_safety_in_re_65ba90b6,
  title = {World Models for Predictive Safety in Reinforcement Learning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/world-models-for-predictive-safety-in-reinforcement-learning/rdr_65ba90b63ac142b7be644aee4bacf02c},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_world_models_for_predictive_safety_in_re_65ba90b6_v1,
  title = {World Models for Predictive Safety in Reinforcement Learning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/world-models-for-predictive-safety-in-reinforcement-learning/rdr_65ba90b63ac142b7be644aee4bacf02c/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}