Applications of Koopman Mode Analysis to Neural Networks

Iva Manojlovic, Maria Fonoberova, Ryan Mohr, Aleksandr Andrejcuk, Zlatko Drmac, Yannis G. Kevrekidis, Igor Mezic

Abstract

We consider the training process of a neural network as a dynamical system acting on the high-dimensional weight space. Each epoch is an application of the map induced by the optimization algorithm and the loss function. Using this induced map, we can apply observables on the weight space and measure their evolution. The evolution of the observables are given by the Koopman operator associated with the induced dynamical system. We use the spectrum and modes of the Koopman operator to realize the above objectives. Our methods can help to, a priori, determine the network depth; determine if we have a bad initialization of the network weights, allowing a restart before training too long; speeding up the training time. Additionally, our methods help enable noise rejection and improve robustness. We show how the Koopman spectrum can be used to determine the number of layers required for the architecture. Additionally, we show how we can elucidate the convergence versus non-convergence of the training process by monitoring the spectrum, in particular, how the existence of eigenvalues clustering around 1 determines when to terminate the learning process. We also show how using Koopman modes we can selectively prune the network to speed up the training procedure. Finally, we show that incorporating loss functions based on negative Sobolev norms can allow for the reconstruction of a multi-scale signal polluted by very large amounts of noise.

Introduction

Despite the transformative success of deep learning, neural networks are frequently criticized as "black boxes." The design of their architectures—such as determining the optimal number of layers—and the management of their training processes often rely on heuristic methods, intuition, and significant computational trial-and-error. This research addresses these challenges by reframing the training of a neural network as a trajectory within a high-dimensional discrete dynamical system.

By viewing each training epoch as an application of a mathematical map induced by the optimizer and the loss function, we can leverage powerful tools from dynamical systems theory. Specifically, we apply Koopman Mode Analysis to "peek under the hood" of the training process, providing a more rigorous framework for architectural decisions and convergence monitoring.

Methodology: The Koopman Operator and DMD

The central challenge in analyzing neural network training is its inherent non-linearity and high dimensionality. Koopman Mode Analysis bypasses this by focusing on the evolution of "observables" (functions of the weights, such as the loss or the weights themselves) rather than the non-linear weight updates directly.

This evolution is governed by the Koopman Operator, a linear but infinite-dimensional operator. To make this computationally tractable, we use Dynamic Mode Decomposition (DMD), specifically the DMD_RRR algorithm, to approximate the Koopman eigenvalues and modes from the training data. This allows us to decompose the complex training dynamics into a set of linear modes, each associated with a specific eigenvalue that describes its growth, decay, or oscillation.

Monitoring Convergence via Spectral Analysis

One of the most significant insights of this work is the use of the Koopman spectrum as a "progress bar" for training. Traditional training monitoring relies on the loss curve, which can often plateau or fluctuate in ways that make it difficult to determine the ideal stopping point.

Standard cross-entropy loss curve for the training process, with an inset showing the fine-grained fluctuations in later epochs.

In contrast, by tracking the eigenvalues of the Koopman operator, we can observe the system's approach to a fixed point. As training progresses and the network nears convergence, the eigenvalues tend to cluster around 1 on the unit circle in the complex plane.

Evolution of Koopman eigenvalues over 500 training epochs, showing clustering behavior as the system approaches a fixed point.

This spectral monitoring provides a principled "early stopping" criterion. If eigenvalues fail to cluster or show unstable trajectories early on, it may indicate a poor weight initialization, allowing for a restart before wasting excessive computational resources.

Architectural Sizing and Efficient Pruning

Beyond monitoring, Koopman analysis informs the very structure of the network. By performing spectral analysis on the input signal, we can analytically determine the number of layers required for specific models, such as Hierarchical Support Vector Regression (HSVR), to capture all relevant scales of the data.

Furthermore, the Koopman modes identify which weights contribute most to the long-term evolution of the training process. This allows for a highly efficient pruning strategy. Unlike standard methods that simply remove the smallest weights, our Koopman-based approach can pin up to 75% of network weights to zero while maintaining or even improving performance relative to the original unpruned network.

Robustness and Noise Rejection

The paper also introduces the use of negative Sobolev norms in loss functions. While standard $L^2$ loss often fails by "averaging" signals buried in high levels of noise, Sobolev norms can be tuned to prioritize coarse-scale features. This spectral tuning makes the network remarkably robust, enabling the reconstruction of underlying multiscale signals even when they are heavily polluted by noise.

Conclusion and Future Work

Treating neural network training as a dynamical system opens new avenues for rigorous analysis and optimization. While currently demonstrated on MNIST-scale architectures, the potential for Koopman Mode Analysis to automate architecture design and provide deeper insights into generalization remains a promising area for future research. Scaling these methods to modern large-scale models will be a key challenge for the next phase of this work.

Summary updated

Abstract

We consider the training process of a neural network as a dynamical system acting on the high-dimensional weight space. Each epoch is an application of the map induced by the optimization algorithm and the loss function. Using this induced map, we can apply observables on the weight space and measure their evolution. The evolution of the observables are given by the Koopman operator associated with the induced dynamical system. We use the spectrum and modes of the Koopman operator to realize the above objectives. Our methods can help to, a priori, determine the network depth; determine if we have a bad initialization of the network weights, allowing a restart before training too long; speeding up the training time. Additionally, our methods help enable noise rejection and improve robustness. We show how the Koopman spectrum can be used to determine the number of layers required for the architecture. Additionally, we show how we can elucidate the convergence versus non-convergence of the training process by monitoring the spectrum, in particular, how the existence of eigenvalues clustering around 1 determines when to terminate the learning process. We also show how using Koopman modes we can selectively prune the network to speed up the training procedure. Finally, we show that incorporating loss functions based on negative Sobolev norms can allow for the reconstruction of a multi-scale signal polluted by very large amounts of noise.
BibTeX
@inproceedings{manojlovic2021applications,
  author = {Iva Manojlovic and Maria Fonoberova and Ryan Mohr and Aleksandr Andrejcuk and Zlatko Drmac and Yannis G. Kevrekidis and Igor Mezic},
  title = {{Applications of Koopman Mode Analysis to Neural Networks}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  url = {https://openreview.net/forum?id=_OOzW4y3PLl},
}

Loading related content...

Figure