Successive Convexification in Learned World Models for Safe Offline Planning

Samuel YoungerarXiv 2026 · arXiv

TL;DR

This paper introduces a novel approach to safe offline planning by replacing traditional sampling-based methods with Successive Convex Programming (SCP) in the latent space of a world model. By enforcing strict regularity constraints on neural networks and using learned Control Barrier Functions, the authors create a planner that treats safety as a hard constraint, outperforming standard samplers in safety and return at long horizons.

What Problem Does This Paper Address?

In offline reinforcement learning, an agent must learn to act safely using only a fixed dataset, without any further interaction with the environment. Most current "world model" planners rely on sampling-based methods like the Cross-Entropy Method (CEM) or Model Predictive Path Integral (MPPI).

The problem? These methods handle safety constraints passively. They sample thousands of trajectories and then "reject" or "penalize" the ones that crash. As the planning horizon grows or the safe path narrows, the probability of randomly sampling a safe trajectory drops precipitously. The safety constraint doesn't guide the search; it only filters the results. Furthermore, these samplers can't tell you if a safe path even exists—they just give you the "least bad" option they happened to find.

Methods

The authors propose a shift from sampling to trajectory optimization via Successive Convexification.

  1. Successive Convex Programming (SCP): Instead of sampling, the planner linearizes the non-convex dynamics and constraints around a reference trajectory. This creates a convex subproblem (a Quadratic Program) that can be solved efficiently. The process iterates until the trajectory converges to a first-order optimal point.
  2. Regularity Constraints: SCP requires smooth dynamics ($C^1$ continuity) and bounded curvature to ensure linearization errors don't explode. To achieve this, the authors build their world model using smoothed MaxMin activations and spectral normalization to enforce Lipschitz bounds on the neural networks.
  3. Learned Control Barrier Functions (CBF): The agent learns a barrier function from the offline data that represents the distance to the nearest safety violation. This is imposed as a hard constraint at every single step of the planning horizon.
  4. Hybrid Initialization (SCP+MPPI): Since SCP is a local optimization method, it can get stuck in poor local minima. The authors warm-start the process with a small number of MPPI samples, using the best sample as the initial "guess" for the SCP solver.

Datasets

The approach was evaluated on the DSRL (Datasets for Safe Reinforcement Learning) benchmarks, specifically:

  • SafetyPointGoal1: A point robot navigating a 2D space with 8 hazards.
  • SafetyPointGoal2: A more difficult task with 10 hazards and 10 "vases." Touching a vase constitutes a safety violation. The models were trained on offline trajectories where any safety violation resulted in an immediate, absorbing failure state.

Key Insights and Ideas

  • Constraint-Guided Search: Unlike sampling, where you hope to hit the safe set by chance, SCP uses the gradients of the safety constraints to actively "push" the trajectory into safe regions of the latent space.
  • The Certifiability Advantage: One of the most significant "aha" moments is that SCP provides a feasibility certificate. At each step, the planner can report whether it found a mathematically optimal (KKT) point, or if the problem is locally infeasible. This is a massive leap over sampling-based methods, which fail silently.
  • Lipschitz Bounds as a Safety Feature: While Lipschitz-constrained networks are often used for robustness, here they are essential for the mathematical validity of the optimization. The authors prove that the linearization error is bounded by the Lipschitz constant and the smoothing width of the activations.

Key Results

  • Superior Safety: In head-to-head comparisons at matched wall-clock time, SCP+MPPI had fewer safety violations than standard MPPI or CEM, even when the samplers were allowed to draw 128 times more samples.
  • Long-Horizon Performance: As the planning horizon ($N$) increased from 10 to 30 steps, the performance gap widened. SCP earned significantly higher returns at $N=20$ and $N=30$, where sampling-based methods struggled to find high-reward paths that remained safe.
  • Reliable Monitoring: The learned barrier function proved to be a highly reliable monitor; 93–98% of actual environment violations were preceded by a plan that the model itself flagged as unsafe.

Limitations and Future Work

  • Model-Environment Mismatch: Safety is enforced within the learned model. If the model is inaccurate (e.g., it "hallucinates" a safe path where none exists), the agent will still fail in the real environment.
  • Computational Overhead: While SCP is more sample-efficient, solving a Quadratic Program (QP) at every iteration is computationally heavier than simple forward passes. The current implementation takes 0.4 to 3.6 seconds per step depending on the horizon.
  • Action Dimensionality: The experiments focused on 2D action spaces. Scaling this to high-dimensional robots (like humanoids) remains a challenge for future research.
  • Fallback Strategies: The authors note that while they can detect "infeasibility," they don't yet have a fallback controller to handle those situations safely.

Summary updated

Abstract

World models learned from offline data allow an agent to plan without further interaction with the environment, but the sampling-based planners used with them handle safety constraints only by scoring or rejecting sampled action sequences. We instead pose safe planning in the latent space of a learned world model as a successive convex program (SCP), in which a learned discrete-time control barrier function is imposed as a constraint at every step of the horizon. SCP requires smooth dynamics, so we build the networks from smoothed MaxMin activations and spectrally normalised layers, and bound the resulting linearisation error in terms of the Lipschitz bound and the smoothing width. As SCP is a local method, we start it from a sampled plan. On two Safety-Gymnasium navigation tasks with offline data, this planner has fewer violations than MPPI and CEM drawing 128 times as many samples at equal wall-clock time in our main comparison, though not significantly, and earns greater return at longer horizons, and is much safer than SCP without a sampled start. SCP converged more often on Lipschitz-bounded models. Unlike sampling, it also reports at each step whether its plan is a feasible, first-order optimal KKT point of the planning problem in the model, or that no feasible plan was found nearby. Our implementation is publicly available at https://github.com/youngers2006/scwm.
BibTeX
@misc{younger2026successive,
  author = {Samuel Younger},
  title = {{Successive Convexification in Learned World Models for Safe Offline Planning}},
  year = {2026},
  eprint = {2611.19466},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.19466},
}
Figure