How Robust are Deep Neural Networks?

Biswa Sengupta, Karl J. FristonarXiv 2018 · arXiv

Abstract

High classification accuracy in Deep Neural Networks (DNNs) is often mistaken for system reliability. This paper demonstrates that learned weight matrices in recurrent networks are frequently "non-normal," meaning they are hyper-sensitive to tiny perturbations. While we can force these networks to be stable through spectral normalization, doing so often makes them more "brittle," highlighting a fundamental trade-off between stability, robustness, and performance.

Introduction: The Robustness Gap

In the development of Recurrent Neural Networks (RNNs), research has long focused on the "vanishing and exploding gradient" problem. Traditionally, the solution has been to manage the spectral radius—ensuring the largest eigenvalue of the weight matrix stays near 1 to prevent information from dissipating or uncontrolled growth over time.

However, this paper argues that mathematical stability (outputs not exploding) does not equate to structural robustness. A system can be stable yet remain hyper-sensitive to minor weight perturbations. This study investigates why highly accurate RNNs, LSTMs, and GRUs remain vulnerable to noise and adversarial attacks, specifically examining the "non-normality" of their weight matrices.

Methodology

The authors evaluate RNN, LSTM, and GRU architectures using tools from linear algebra and dynamical systems theory:

  • Henrici’s Number: A metric for "non-normality." A normal matrix ($W^*W = WW^*$) has a Henrici number of zero; higher values indicate non-orthogonal eigenvectors, making the system sensitive to perturbations.
  • $\epsilon$-pseudospectrum: A qualitative tool mapping how eigenvalues shift under small perturbations ($\epsilon$). This creates a "spectral portrait" of the network's robustness.
  • Spectral Stabilization: Using the Power Method to constrain the spectral radius within the unit disk to re-evaluate robustness after ensuring stability.
  • Schur Decomposition: Separating the weight matrix into its eigenvalues and its strictly upper triangular (non-normal) components.
Spectral portraits (pseudospectra) for 2-layer RNN, LSTM, and GRU architectures.

Experimental Setup and Datasets

The study utilizes three benchmarks to test temporal learning:

  1. Adding Task: Requires the network to remember and sum specific numbers in long sequences.
  2. IMDB Sentiment Analysis: Classifying movie reviews as positive or negative.
  3. Sequential MNIST: Processing digit pixels one by one to test long-term memory.

Key Insights: The Nature of Non-Normality

  • Accuracy $\neq$ Robustness: Networks can achieve 99% accuracy while possessing highly non-normal and unstable weight matrices. High performance is not a proxy for safety.
  • Non-normality is the Norm: Training via backpropagation naturally results in non-normal matrices where eigenvectors may not be orthogonal, leading to transient growth and instability.
  • The Edge of Criticality: To "remember" long sequences, Jacobian eigenvalues must approach the unit circle. This "edge of chaos" allows for high information processing but results in extreme fragility.

Key Results: The Stability-Robustness Paradox

The paper's most counter-intuitive finding is the "brittle" nature of normalization. When spectral normalization was used to force stability, the $\epsilon$-pseudospectrum showed that the networks actually became more brittle. Tiny perturbations caused significantly larger shifts in the spectral portrait than in un-normalized networks.

Detailed pseudospectra of internal gate matrices (i, f, c, o) for a 2-layer LSTM.

Quantitatively, significant Henrici numbers were found across all tasks. In the IMDB task, LSTMs and GRUs showed high non-normality across internal gates. While RNNs and GRUs performed well on Sequential MNIST (19-26% accuracy), LSTMs often failed to converge, highlighting the difficulty of training these complex dynamical systems.

Conclusion and Future Directions

Constraining the eigen-spectrum is insufficient for safety-critical AI. The authors propose two paths forward:

  1. Simultaneous Optimization: Developing algorithms that optimize for performance, stability, and robustness concurrently using pseudospectral tools.
  2. The Variational Approach: Leveraging the Free Energy Principle and Variational Bayes to find a balance between complexity and accuracy, potentially leading to solutions that are inherently more robust to noise.

Until the underlying linear algebra of weight matrices is addressed, deep learning systems will remain fundamentally brittle despite their surface-level accuracy.

Summary updated

Abstract

Convolutional and Recurrent, deep neural networks have been successful in machine learning systems for computer vision, reinforcement learning, and other allied fields. However, the robustness of such neural networks is seldom apprised, especially after high classification accuracy has been attained. In this paper, we evaluate the robustness of three recurrent neural networks to tiny perturbations, on three widely used datasets, to argue that high accuracy does not always mean a stable and a robust (to bounded perturbations, adversarial attacks, etc.) system. Especially, normalizing the spectrum of the discrete recurrent network to bound the spectrum (using power method, Rayleigh quotient, etc.) on a unit disk produces stable, albeit highly non-robust neural networks. Furthermore, using the $ε$-pseudo-spectrum, we show that training of recurrent networks, say using gradient-based methods, often result in non-normal matrices that may or may not be diagonalizable. Therefore, the open problem lies in constructing methods that optimize not only for accuracy but also for the stability and the robustness of the underlying neural network, a criterion that is distinct from the other.
BibTeX
@misc{sengupta2018how,
  author = {Biswa Sengupta and Karl J. Friston},
  title = {{How Robust are Deep Neural Networks?}},
  year = {2018},
  eprint = {1804.11313},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1804.11313},
}
Figure