When deploying neural networks in safety-critical feedback loops—such as a recurrent neural network (RNN) acting as a flight controller—we need rigorous mathematical guarantees that the system will remain stable under perturbations. Traditionally, control theorists evaluate system stability using the Small-Gain Theorem. This theorem relies on induced norms ($L_2$ or $l_2$), which measure the maximum possible amplification (or gain) of a signal passing through a system. If the loop's total worst-case gain is strictly less than one, the system is guaranteed to be stable.
However, standard induced norms assume that input signals can be anything—positive or negative. As pointed out by Yoshio Ebihara et al. in l2 induced norm analysis of discrete-time LTI systems for nonnegative input signals and its application to stability analysis of recurrent neural networks, this assumption introduces severe conservatism when analyzing neural networks. Modern deep networks rely heavily on activation functions like the Rectified Linear Unit (ReLU), which clip negative values and output only nonnegative signals. By evaluating worst-case amplification over all real-valued signals, standard control-theoretic tools test the system against physically impossible negative inputs. This results in overly pessimistic stability bounds, flagging perfectly safe networks as potentially unstable.
Restricting the Input Space to the Nonnegative Orthant
To build more accurate, less conservative safety guarantees, a growing body of research focuses on redefining the core metric of signal amplification. Instead of measuring the standard $L_p$ norm, the authors propose evaluating the nonnegative induced norm ($L_p^+$ or $l_p^+$).
As formalized by Yoshio Ebihara et al. in Induced norm analysis of linear systems for nonnegative input signals, the $L_p^+$ norm restricts the mathematical search for the worst-case input signal strictly to the nonnegative orthant ($u(t) \ge 0$).
Standard L2 Norm Search Space: Nonnegative L2+ Norm Search Space:
▲ ▲
│ │
◄─────┼─────► ──────┼─────►
│ │
▼ ▼
(All real signals) (Only positive signals)
By ignoring negative inputs, this specialized norm accurately captures the true behavior of systems bounded by one-sided nonlinearities. If a ReLU activation function guarantees that a signal never drops below zero, the $L_2^+$ norm calculates the maximum amplification solely for these valid, positive trajectories. This exclusion of impossible failure modes allows researchers to certify much larger regions of guaranteed stability in recurrent architectures.
From Semidefinite to Copositive Programming
If restricting the input space yields tighter bounds, why hasn't this been the standard approach all along? The challenge lies in the mathematics of optimization.
Standard $L_2$ norm computation relies on Linear Matrix Inequalities (LMIs) and positive semidefinite (PSD) matrices, which can be solved efficiently. A matrix $M$ is PSD if its quadratic form is nonnegative for all possible vectors ($x^T M x \ge 0$).
When we restrict our attention to nonnegative inputs, we no longer require $x^T M x \ge 0$ everywhere—only where the vector $x$ is element-wise nonnegative ($x \ge 0$). This shifts the problem from standard semidefinite programming to copositive programming. A matrix is copositive if its quadratic form is nonnegative specifically on the nonnegative orthant. While copositive programming allows for much tighter bounds, it is computationally harder to evaluate exactly.
To bridge this gap, Yoshio Ebihara et al. integrated this concept into the Integral Quadratic Constraint (IQC) framework in Stability Analysis of Recurrent Neural Networks by IQC with Copositive Mutipliers. By replacing traditional PSD multipliers with copositive multipliers, the authors successfully constrained the stability analysis to match the physical constraints of ReLU activations.
To further refine these calculations—particularly for challenging single-input systems where previous methods yielded only trivial bounds—Yoshio Ebihara et al. introduced the use of "positive filters" in L2+ Induced Norm Analysis of Continuous-Time LTI Systems Using Positive Filters and Copositive Programming. These filters provide a scalable way to compute tight upper and lower bounds for continuous-time linear time-invariant (LTI) systems processing nonnegative signals.
Direct Nonlinear Verification via Sum of Squares
While computing specialized induced norms for the linear components of a network is highly effective, another complementary approach is to analyze the nonlinear feedback loop as a whole. Instead of abstracting the neural network's activations into a restricted norm, one can directly model the algebraic equations of the nonlinearities.
In Stability of Non-linear Neural Feedback Loops using Sum of Squares, Matthew Newton et al. demonstrate that Sum of Squares (SOS) programming can directly ingest the nonlinear equations of a neural network controller within a feedback loop. This method bypasses conservative sector-bounding techniques entirely. By translating the network's exact activation behavior into polynomial constraints, the authors can rigorously verify the system's stability and compute its precise region of attraction, even when the underlying physical plant dynamics are nonlinear.
Conclusion
Whether through the lens of specialized $L_2^+$ induced norms or direct SOS verification, the core insight driving this research direction is clear: traditional control theory is overly pessimistic because it prepares for impossible inputs. By tailoring mathematical tools—via copositive multipliers, positive filters, and restricted norms—to respect the inherent non-negativity of neural network activations, researchers can unlock significantly more precise and realistic safety guarantees for modern autonomous systems.
Key Foundational Work
l2 induced norm analysis of discrete-time LTI systems for nonnegative input signals and its application to stability analysis of recurrent neural networks (2021) — contributed 7 concept elements to this direction.
Stability Analysis of Recurrent Neural Networks by IQC with Copositive Mutipliers (2022) — contributed 4 concept elements to this direction.
L2+ Induced Norm Analysis of Continuous-Time LTI Systems Using Positive Filters and Copositive Programming (2022) — contributed 2 concept elements to this direction.