LyEvO: Lyapunov-Guided Evolutionary Optimization for Safe and Robust Sim-to-Real Policy Learning

Riccardo Curcio, Hongpeng Cao, Marco CaccamoarXiv 2026 · arXiv

TL;DR

LyEvO is a new physics-grounded framework that bridges the gap between safe policy learning and real-world deployment readiness. By combining evolutionary optimization, statistical model checking, and Lyapunov stability analysis, it systematically expands and certifies a neural network controller's safe operating region. The result is a control policy equipped with formal statistical guarantees, ensuring robust and safe sim-to-real transfer.

What Problem Does This Paper Address?

Deep Reinforcement Learning (DRL) has shown incredible promise in training robotic controllers within simulation. However, deploying these controllers onto real hardware remains notoriously risky due to the sim-to-real gap—modeling mismatches and unpredictable real-world disturbances can cause a policy that performed perfectly in simulation to fail catastrophically in reality.

Existing methods attempt to tackle this via safe RL (adding safety penalties or constraints during training) or robust RL (domain randomization). However, these approaches typically lack formal guarantees: they cannot precisely tell an engineer where (under what initial conditions or disturbance levels) the controller is guaranteed to remain safe, or with what level of confidence. Conversely, formal verification methods can provide guarantees but are usually applied post-hoc to a fixed policy, completely disconnected from the training process.

This paper introduces LyEvO (Lyapunov-Guided Evolutionary Optimization) to unify policy synthesis, safety verification, and operational region estimation into a single, iterative loop.

Methods

LyEvO uses a physics-grounded, bilevel optimization approach to jointly optimize a policy and maximize its verified safe operating domain.

  1. Lyapunov-Based Initialization: The framework starts with a nominal linear model of the system and uses Lyapunov’s direct method (solved via Convex Optimization/Linear Matrix Inequalities) to compute an initial, mathematically certified ellipsoidal stability region ($\Omega$).
  2. Outer Loop (Region Expansion): LyEvO progressively expands this ellipsoid along its principal directions using a greedy heuristic. This systematically increases the range of initial conditions the controller can safely recover from.
  3. Inner Loop (Policy Search & Verification): For each expanded region, LyEvO runs an algorithm called SMC-ES (Statistical Model Checking - Evolution Strategies). This counterexample-driven loop uses Evolution Strategies to optimize the neural network policy parameters over the current region. Simultaneously, it uses Statistical Model Checking (SMC) to sample scenarios and verify if the policy meets both safety constraints and a performance lower bound ($\beta^*$).

If verification passes, the region expands further. If it fails, the discovered counterexamples are added to the training set, the region rolls back, and the boundaries are marked as fixed. Upon convergence, the policy receives a formal certificate: with at least $1-\delta$ confidence, the probability of encountering a failure scenario within the expanded region is at most $\varepsilon$.

Fig. 1: Illustration of the initial and enlarged Cartpole stability regions in a 2D projection, where sampled states with larger Lyapunov values lie closer to the region boundary.

Datasets

The framework was evaluated on two classic nonlinear control benchmarks using both simulation and physical hardware:

  • Cartpole: A 4D state space system where the cart must track a position while balancing a rigid pole upright. Real-world testing was done on a physical Quanser platform.
  • 3D Quadrotor: A highly challenging 12D state space goal-reaching task with low-level thrust control (4 rotor inputs). Simulation was conducted in PyBullet and Gazebo, and real-world deployment was validated in an indoor flight cage using an ANT-X quadrotor.

Key Insights and Ideas

  • Geometry-Guided Sampling: Instead of uniformly or randomly sampling initial states (which can lead to uninformative or physically impossible training scenarios), LyEvO leverages the geometric structure of a Lyapunov ellipsoid. States with higher Lyapunov values lie closer to the boundary, allowing the algorithm to intelligently test the controller's limits.
  • Unified Performance and Safety Certificates: Unlike standard safe RL methods that treat safety and task performance as competing objectives, LyEvO's verification protocol treats them as a unified predicate. A scenario is a failure if the robot violates a safety boundary or if its cumulative reward drops below a guaranteed threshold.
  • Robustness via Noise Injection: A variant of the framework, LyEvO-R, injects observation and action noise during the evolutionary optimization phase. This forces the policy to adopt a smoother, more conservative control strategy that translates exceptionally well to real hardware.

Key Results

  • Massive Region Expansion: In simulation, LyEvO successfully expanded the verified safe operating region to be 6.14× larger for the Cartpole and 224.04× larger for the 3D Quadrotor compared to standard baseline certified regions.
  • Flawless Safety Verification: Across both standard and enlarged regions, LyEvO and LyEvO-R achieved a 0% Violation Probability (VP) under formal verification. In contrast, standard DRL baselines (like SAC, SAC-Lagrangian, and SAC-Lyapunov) failed verification completely, exhibiting violation probabilities as high as 54%.
  • Real-World Resilience: When deployed on physical hardware under noisy and disturbed conditions (such as blasting the quadrotor with a high-power fan), LyEvO-R achieved the smoothest trajectories and lowest tracking errors, outperforming both standard LyEvO and traditional baseline mathematical controllers (LMI).
Fig. 3: Trajectory comparison for different policies on Cartpole system in real world under noise.

Limitations and Future Work

  • Computational Cost: Evolution Strategies are notoriously sample-inefficient compared to gradient-based DRL. LyEvO required parallelization across up to 1,000 CPU cores, taking roughly 14.3 hours for the nominal quadrotor and 31.5 hours for the robust version (LyEvO-R).
  • Actuation Jitter: The learned neural network policies exhibit more high-frequency actuation jitter than traditional linear controllers, which could accelerate mechanical wear on hardware.
  • Linear Prior Dependence: The initialization step currently relies on a nominal linearized system model to generate the initial Lyapunov matrix. The authors note that future work will focus on extending this framework to highly nonlinear dynamics where a clean linear prior isn't readily available.

Summary updated

Abstract

Training controllers that are safe and robust in simulation, and systematically assessing their readiness for real-world deployment, remain key challenges in sim-to-real transfer. To address this, we propose LyEvO, a physics-grounded framework that combines constrained Evolutionary Optimization and Statistical Model Checking (SMC)-based verification with Lyapunov-based stability analysis. Leveraging prior knowledge of the system dynamics, LyEvO uses Lyapunov analysis to compute an initial candidate stability region. An iterative loop then uses operational scenarios drawn from this region to jointly optimize and statistically verify a policy, and subsequently expands the region's boundaries based on the verification outcome. This integrated procedure provides a practical criterion for assessing deployment readiness. We evaluate LyEvO on Cartpole and 3D Quadrotor benchmarks through extensive simulations and targeted real-world experiments, demonstrating safe and robust sim-to-real transfer.
BibTeX
@misc{curcio2026lyevo,
  author = {Riccardo Curcio and Hongpeng Cao and Marco Caccamo},
  title = {{LyEvO: Lyapunov-Guided Evolutionary Optimization for Safe and Robust Sim-to-Real Policy Learning}},
  year = {2026},
  eprint = {2608.06481},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2608.06481},
}
Figure