Runtime Safety Assurance for Neural Network Controllers

This direction is merged into another direction: formal safety verification (formal stability guarantees). Its last version stays readable and citable here.

Version 1 of 116 papers16 conceptsfrozen 2026-10-04

Modern autonomous systems face a fundamental tension: the most capable controllers are often the least trustworthy. Deep Reinforcement Learning (DRL) and neural networks offer unparalleled performance in complex, high-dimensional tasks like navigating mixed-autonomy traffic Cathy Wu or managing aircraft separation. However, these models are "black boxes." As noted by Benedikt Maderbacher and Majid Zamani, their opaque nature makes traditional formal verification—proving they will never fail—nearly impossible.

To bridge this gap, researchers are developing Runtime Safety Assurance (RSA) frameworks. Instead of trying to prove the neural network is perfect, RSA treats the AI as an untrusted agent and wraps it in a "safety net" that monitors and corrects its actions in real-time.

The Simplex Architecture: A Safety Net for AI

The foundational concept for this research is the Simplex Architecture. As reviewed by Qiang Chen, Simplex splits the control system into three distinct parts:

  1. High-Performance Controller (HPC): An unverified, complex AI (like a neural network) that handles the primary mission.
  2. Safety Controller (SC): A simple, rigorously verified backup—often based on traditional Model Predictive Control—that knows how to keep the system in a "safe zone."
  3. Decision Logic: A supervisor that monitors the state. If the HPC's proposed action would push the system toward a crash, the logic instantly switches control to the SC.

The intuition is simple: you don't need to understand why the AI is making a mistake; you only need to know when the system is approaching the boundary of its "safe invariant" set.

Sandboxing the Black Box

Recent advancements have focused on making this "switching" more efficient and less intrusive. Renzhi Wang and Lei Ma introduced MORTAR, a framework that repairs unsafe actions with less than 1.5 milliseconds of delay. By treating the AI as a black box, MORTAR can "patch" dangerous commands on the fly without needing to retrain the underlying model.

Similarly, Bingzhuo Zhong and Marco Caccamo proposed the Safe-visor architecture. This acts as a sandbox for unverified controllers in stochastic (noisy) environments. By using mathematical abstractions of the physical system, the Safe-visor provides a formal guarantee that the system remains safe even if the AI makes "harmful" decisions.

Probabilistic Guarantees in a Messy World

In the real world, safety isn't always binary. Sensors have noise, and human drivers are unpredictable. Kyle D. Julian and Mykel J. Kochenderfer have moved beyond "safe/unsafe" labels toward probabilistic safety guarantees. By combining formal verification with Markov Decision Process (MDP) model checking, they can quantify risk—for example, proving that an autonomous aircraft has a less than 5% chance of collision in a specific stochastic environment.

This probabilistic approach is critical for "mixed autonomy," a term championed by Cathy Wu in the Flow framework. In these scenarios, autonomous vehicles must coordinate with human-driven cars. Because human behavior cannot be perfectly predicted, the safety net must account for uncertainty while still allowing the AI enough freedom to optimize traffic flow.

Expanding the Safe Zone

A common criticism of safety backups is that they are too conservative, often overriding the AI and hurting performance. To solve this, Benedikt Maderbacher and Roderick Bloem developed an adaptive Simplex architecture.

Instead of using a fixed safety zone, their system uses "proofs on demand" to dynamically expand the safe operating area at runtime. If the AI wants to perform a maneuver that was previously considered "risky," the system attempts to quickly verify a new, larger safe zone. This ensures that the safety net only intervenes when absolutely necessary, maximizing the performance benefits of the neural network controller.

The Path Forward: Compositional Safety

As autonomous missions become more complex—involving multiple robots or long-term logic tasks—researchers like Jun Wang are looking at compositional neuro-symbolic control. This involves breaking a mission down into smaller "skills" (each controlled by a neural network) and using automata theory to stitch them together safely.

By verifying the transitions between these skills, we can build systems that are both highly capable and formally reliable, moving us past the current "bottleneck period" of autonomous driving identified by Yuchen Li and Yang Xing.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_runtime_safety_assurance_for_neural_netw_da8de320,
  title = {Runtime Safety Assurance for Neural Network Controllers},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/runtime-safety-assurance-for-neural-network-controllers/rdr_da8de320a1b54d21938ddac363633566},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_runtime_safety_assurance_for_neural_netw_da8de320_v1,
  title = {Runtime Safety Assurance for Neural Network Controllers},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/runtime-safety-assurance-for-neural-network-controllers/rdr_da8de320a1b54d21938ddac363633566/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}