Applications of Koopman Mode Analysis to Neural Networks

Ryan Mohr, Maria Fonoberova, Iva Manojlovic, Aleksandr Andrejcuk, Zlatko Drmac, Yannis G. Kevrekidis, Igor Mezic

Abstract

Traditional neural network training often lacks transparency regarding internal dynamics. This work treats training as a high-dimensional dynamical system, applying Koopman operator theory to monitor convergence in real-time. Additionally, we introduce negative Sobolev loss functions to enhance multi-scale signal reconstruction under extreme noise conditions.

Introduction

Neural network training is typically viewed through the lens of black-box optimization. While loss curves provide a macroscopic view of progress, they do not reveal the underlying "health" of the training dynamics. This paper addresses two critical challenges: the lack of rigorous tools for monitoring internal training states and the failure of standard $L^2$ loss functions to handle multi-scale signals polluted by high-frequency noise.

Methodology

The proposed approach integrates dynamical systems theory with deep learning via two main techniques.

Koopman Operator and Dynamic Mode Decomposition

We model the weight update rule, $w_{t+1} = T(w_t)$, as a discrete dynamical system. Given its high dimensionality and nonlinearity, we utilize the Koopman operator—a linear but infinite-dimensional operator that describes the evolution of observables. We approximate this operator using Dynamic Mode Decomposition with Rank-Revealing QR (DMD_RRR).

Negative Sobolev Loss Functions

To improve robustness against noise, we move beyond Mean Squared Error. We propose loss functions based on negative-index Sobolev norms ($H^{-s}$). By using Fourier transforms to weight errors, these norms prioritize large-scale features (low frequency) and reduce sensitivity to small-scale discrepancies (high-frequency noise), effectively acting as an integrated low-pass filter.

Monitoring Training Dynamics

The eigenvalues of the Koopman operator serve as a diagnostic tool for the training state. Early in the process, eigenvalues clustered near zero indicate rapid learning and significant weight adjustments. As training stabilizes and approaches a fixed point, these eigenvalues migrate toward the unit circle.

Koopman operator eigenvalues after 40 epochs of training on MNIST, showing the initial learning phase.

Experimental Results

The methodology was evaluated using the MNIST dataset for spectral analysis and synthetic multi-scale signals for noise robustness testing.

Convergence Monitoring

On MNIST, the Koopman spectrum successfully tracked the transition from rapid learning to stabilization. By epoch 500, the eigenvalues clearly clustered around the unit circle, confirming that the system was nearing convergence.

Eigenvalues after 500 epochs, demonstrating clustering around the unit circle as the network reaches a stable state.

Noise Robustness

In multi-scale signal reconstruction, standard $L^2$ loss failed as noise increased, capturing only the average of the noisy signal. In contrast, the "Sobolev loss 1" approach, which applies layer-wise scaling, maintained high reconstruction accuracy even when the noise magnitude equaled the signal magnitude ($\epsilon = 1.0$).

Conclusion

Koopman Mode Analysis provides a powerful framework for real-time monitoring of neural network training. While the Sobolev loss functions offer superior noise rejection, future research is needed to scale these methods to modern large-scale architectures and structured real-world noise.

Summary updated

Abstract

We consider the training process of a neural network as a dynamical system acting on the high-dimensional weight space. Each epoch is an application of the map induced by the optimization algorithm and the loss function. Using this induced map, we can apply observables on the weight space and measure their evolution. The evolution of the observables are given by the Koopman operator associated with the induced dynamical system. We use the spectrum and modes of the Koopman operator to realize the above objectives. Our methods can help to, a priori, determine the network depth; determine if we have a bad initialization of the network weights, allowing a restart before training too long; speeding up the training time. Additionally, our methods help enable noise rejection and improve robustness. We show how the Koopman spectrum can be used to determine the number of layers required for the architecture. Additionally, we show how we can elucidate the convergence versus non-convergence of the training process by monitoring the spectrum, in particular, how the existence of eigenvalues clustering around 1 determines when to terminate the learning process. We also show how using Koopman modes we can selectively prune the network to speed up the training procedure. Finally, we show that incorporating loss functions based on negative Sobolev norms can allow for the reconstruction of a multi-scale signal polluted by very large amounts of noise.
BibTeX
@inproceedings{mohr2021applications,
  author = {Ryan Mohr and Maria Fonoberova and Iva Manojlovic and Aleksandr Andrejcuk and Zlatko Drmac and Yannis G. Kevrekidis and Igor Mezic},
  title = {{Applications of Koopman Mode Analysis to Neural Networks}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  url = {https://openreview.net/forum?id=_OOzW4y3PLl},
}

Loading related content...

Figure