TL;DR
This paper introduces a novel approach to safe offline planning by replacing traditional sampling-based methods with Successive Convex Programming (SCP) in the latent space of a world model. By enforcing strict regularity constraints on neural networks and using learned Control Barrier Functions, the authors create a planner that treats safety as a hard constraint, outperforming standard samplers in safety and return at long horizons.
What Problem Does This Paper Address?
In offline reinforcement learning, an agent must learn to act safely using only a fixed dataset, without any further interaction with the environment. Most current "world model" planners rely on sampling-based methods like the Cross-Entropy Method (CEM) or Model Predictive Path Integral (MPPI).
The problem? These methods handle safety constraints passively. They sample thousands of trajectories and then "reject" or "penalize" the ones that crash. As the planning horizon grows or the safe path narrows, the probability of randomly sampling a safe trajectory drops precipitously. The safety constraint doesn't guide the search; it only filters the results. Furthermore, these samplers can't tell you if a safe path even exists—they just give you the "least bad" option they happened to find.
Methods
The authors propose a shift from sampling to trajectory optimization via Successive Convexification.
- Successive Convex Programming (SCP): Instead of sampling, the planner linearizes the non-convex dynamics and constraints around a reference trajectory. This creates a convex subproblem (a Quadratic Program) that can be solved efficiently. The process iterates until the trajectory converges to a first-order optimal point.
- Regularity Constraints: SCP requires smooth dynamics ($C^1$ continuity) and bounded curvature to ensure linearization errors don't explode. To achieve this, the authors build their world model using smoothed MaxMin activations and spectral normalization to enforce Lipschitz bounds on the neural networks.
- Learned Control Barrier Functions (CBF): The agent learns a barrier function from the offline data that represents the distance to the nearest safety violation. This is imposed as a hard constraint at every single step of the planning horizon.
- Hybrid Initialization (SCP+MPPI): Since SCP is a local optimization method, it can get stuck in poor local minima. The authors warm-start the process with a small number of MPPI samples, using the best sample as the initial "guess" for the SCP solver.
Datasets
The approach was evaluated on the DSRL (Datasets for Safe Reinforcement Learning) benchmarks, specifically:
- SafetyPointGoal1: A point robot navigating a 2D space with 8 hazards.
- SafetyPointGoal2: A more difficult task with 10 hazards and 10 "vases." Touching a vase constitutes a safety violation. The models were trained on offline trajectories where any safety violation resulted in an immediate, absorbing failure state.
Key Insights and Ideas
- Constraint-Guided Search: Unlike sampling, where you hope to hit the safe set by chance, SCP uses the gradients of the safety constraints to actively "push" the trajectory into safe regions of the latent space.
- The Certifiability Advantage: One of the most significant "aha" moments is that SCP provides a feasibility certificate. At each step, the planner can report whether it found a mathematically optimal (KKT) point, or if the problem is locally infeasible. This is a massive leap over sampling-based methods, which fail silently.
- Lipschitz Bounds as a Safety Feature: While Lipschitz-constrained networks are often used for robustness, here they are essential for the mathematical validity of the optimization. The authors prove that the linearization error is bounded by the Lipschitz constant and the smoothing width of the activations.
Key Results
- Superior Safety: In head-to-head comparisons at matched wall-clock time, SCP+MPPI had fewer safety violations than standard MPPI or CEM, even when the samplers were allowed to draw 128 times more samples.
- Long-Horizon Performance: As the planning horizon ($N$) increased from 10 to 30 steps, the performance gap widened. SCP earned significantly higher returns at $N=20$ and $N=30$, where sampling-based methods struggled to find high-reward paths that remained safe.
- Reliable Monitoring: The learned barrier function proved to be a highly reliable monitor; 93–98% of actual environment violations were preceded by a plan that the model itself flagged as unsafe.
Limitations and Future Work
- Model-Environment Mismatch: Safety is enforced within the learned model. If the model is inaccurate (e.g., it "hallucinates" a safe path where none exists), the agent will still fail in the real environment.
- Computational Overhead: While SCP is more sample-efficient, solving a Quadratic Program (QP) at every iteration is computationally heavier than simple forward passes. The current implementation takes 0.4 to 3.6 seconds per step depending on the horizon.
- Action Dimensionality: The experiments focused on 2D action spaces. Scaling this to high-dimensional robots (like humanoids) remains a challenge for future research.
- Fallback Strategies: The authors note that while they can detect "infeasibility," they don't yet have a fallback controller to handle those situations safely.
Summary updated