Introduction
Deep Reinforcement Learning (DRL) has revolutionized complex system control, yet these models remain notoriously fragile "black boxes." Verifying DRL systems is significantly more challenging than standard image classifiers because it involves sequential decision-making. A single action by a DRL agent alters the environment, which subsequently changes the next input the agent receives.
Existing verification tools often hit a "scalability wall." While they can identify specific bugs (a "SAT" result), they struggle to prove that a system is always safe over infinite executions (an "UNSAT" result). Previous iterations, such as whiRL 1.0, relied on Bounded Model Checking (BMC), which only examines a limited number of future steps. If a failure occurs beyond that horizon, BMC misses it, failing to provide a comprehensive safety guarantee.
Methodology: The whiRL 2.0 Framework
The whiRL 2.0 tool introduces a scalable framework for formally verifying DRL systems by moving beyond simple bug-finding to providing actual proofs of safety and liveness. It introduces three primary enhancements to the verification pipeline:
- $k$-Induction: This serves as the core engine for proving properties over infinite horizons. Instead of checking from the initial state, $k$-induction analyzes arbitrary sequences of $k$ steps. By showing that a property holds for the first $k$ steps and that its validity for $k$ steps necessitates its validity for the $(k+1)$-th step, a mathematical proof of perpetual safety is established.
- Semi-Automated Invariant Inference: Finding "invariants"—conditions that remain true throughout execution—is critical but often undecidable. whiRL 2.0 exploits the fact that many networking properties are Boolean monotonic functions, using a binary search strategy to automatically identify the "tipping point" where a property transitions from potentially violated to guaranteed.
- Input Abstraction: DRL agents often process a "sliding window" of past observations. whiRL 2.0 can "strip away" irrelevant input fields. If a property holds even when the verifier can pick any value for certain inputs, it proves those inputs do not impact the core safety of the policy.
The tool utilizes Marabou, an SMT-based DNN verifier, as its underlying solving engine to handle the complex constraints of neural network architectures.
Evaluation and Case Studies
The authors evaluated whiRL 2.0 on three state-of-the-art DRL systems in the communication networking domain: Aurora (congestion control), Pensieve (adaptive bitrate streaming), and DeepRM (cluster resource management). These systems provide clear semantic inputs (e.g., latency, packet loss), making them ideal for formal specification.
Key Results
- Aurora: The framework successfully proved two liveness properties—guarantees that "good" things eventually happen. For instance, it proved that under perfect network conditions, the controller will eventually increase its sending rate. Property 1 was proven in seconds ($k=2$), while Property 2 (rate decrease in poor conditions) took 4.5 hours ($k=5$).
- Pensieve: The tool proved that the system correctly decreases video resolution under poor network conditions in approximately 20 minutes—a property previously considered unprovable.
- DeepRM: Through input abstraction, the authors consolidated three separate safety properties into one, revealing a "suboptimal bias" where the agent favored specific job IDs regardless of resource availability.
Conclusion and Future Work
whiRL 2.0 demonstrates that formal verification can move beyond safety to address liveness and interpretability. By using abstraction, the authors discovered that some controllers only prioritize the most recent observations, even when trained on larger windows.
Despite these advances, challenges remain. Increasing $k$ in induction can lead to exponential computation times, and invariant inference still requires user-provided templates. Future research will focus on automating template discovery and integrating advanced search heuristics to further prune the state space during the $k$-induction process.
Summary updated