Guarded Deep Learning using Scenario-based Modeling

Guy Katz

Introduction

Deep Neural Networks (DNNs) have revolutionized fields ranging from computer vision to resource management, often outperforming human-coded logic. However, their "black box" nature presents significant risks in safety-critical applications. When a DNN behaves unpredictably, engineers often resort to hand-crafted "override rules"—ad-hoc if-then statements designed to bypass the network's decisions. This paper proposes a more robust alternative: Scenario-Based Modeling (SBM), a framework that allows safety guards to be expressed as intuitive, modular, and formally verifiable components.

The Challenge of Opaque Neural Networks

While DNNs are powerful, they suffer from two primary drawbacks that hinder their deployment in sensitive environments:

  1. Unpredictability: Minor perturbations in input data can lead to catastrophic failures, such as an autonomous vehicle failing to recognize a stop sign under specific lighting conditions.
  2. Maintenance Complexity: As systems mature, the collection of hard-coded safety overrides often evolves into "spaghetti code." These tangled logic blocks are nearly impossible to debug, verify, or maintain, creating a technical debt that can compromise system integrity.

Methods: Scenario-Based Guarding

The core innovation of this research is the integration of DNNs into the Scenario-Based Modeling paradigm. In SBM, a system is not a monolithic block of code but a collection of independent "scenario objects." These objects interact through a global execution mechanism using three fundamental idioms: requesting an event, waiting-for an event, or blocking an event.

A simple deep neural network with inputs x, hidden nodes v, and outputs y.

To bridge the gap between neural networks and SBM, the author represents the DNN itself as a non-deterministic scenario, denoted as $O_{DNN}$. This scenario waits for an input and then requests all possible output actions, ranked by the network's internal confidence or preference.

The abstraction of a DNN as a scenario (O_DNN) that waits for inputs and requests all possible output events.

Safety rules are implemented as separate, independent scenarios. When a safety scenario identifies a dangerous state, it simply blocks the DNN’s preferred output. The execution mechanism then automatically selects the DNN's next-best, unblocked alternative. This "next-best" strategy ensures the system retains as much of the learned intelligence as possible while remaining within safety bounds.

Formal Verification and Deadlock Prevention

A significant advantage of using SBM is its rigorous mathematical foundation. By decoupling safety logic from the core DNN, the framework allows for automated analysis using formal methods like SMT (Satisfiability Modulo Theories) solvers and model checking.

This is crucial because as more safety and "liveness" (ensuring the system eventually does something good) rules are added, they may inadvertently conflict. The author demonstrates how formal verification can detect "deadlocks"—states where the combined rules block every possible action the DNN wants to take—before the system is ever deployed.

Evaluation: Resource Management and Network Control

The approach was evaluated on two distinct DNN-based systems:

  • DeepRM (Resource Management): The framework was used to prevent the DNN from allocating CPUs and memory to "phantom" jobs (jobs not actually in the queue). The ad-hoc Python overrides were successfully replaced with clean, readable C++ scenarios.
  • Custard (Network Congestion Control): A liveness rule was implemented to prevent the controller from becoming stagnant. If the DNN's outputs remained identical for ten consecutive rounds, the scenario blocked the current output, forcing the network to explore different bit rates and maintain throughput.

Conclusion

By treating safety guards as first-class, modular components rather than ad-hoc patches, Scenario-Based Modeling provides a path toward "Guarded Deep Learning." This framework not only makes safety rules more human-readable and maintainable but also subjects them to the rigors of formal verification, ensuring that the power of deep learning can be harnessed safely in complex, real-world environments.

Summary updated

BibTeX
@inproceedings{katz2020guarded,
  author = {Guy Katz},
  title = {{Guarded Deep Learning using Scenario-based Modeling}},
  booktitle = {MODELSWARD},
  year = {2020},
  url = {https://openreview.net/forum?id=cFbfS_j14fv},
}

Loading related content...

Figure