Abstract
Traditional neural network training often lacks transparency regarding internal dynamics. This work treats training as a high-dimensional dynamical system, applying Koopman operator theory to monitor convergence in real-time. Additionally, we introduce negative Sobolev loss functions to enhance multi-scale signal reconstruction under extreme noise conditions.
Introduction
Neural network training is typically viewed through the lens of black-box optimization. While loss curves provide a macroscopic view of progress, they do not reveal the underlying "health" of the training dynamics. This paper addresses two critical challenges: the lack of rigorous tools for monitoring internal training states and the failure of standard $L^2$ loss functions to handle multi-scale signals polluted by high-frequency noise.
Methodology
The proposed approach integrates dynamical systems theory with deep learning via two main techniques.
Koopman Operator and Dynamic Mode Decomposition
We model the weight update rule, $w_{t+1} = T(w_t)$, as a discrete dynamical system. Given its high dimensionality and nonlinearity, we utilize the Koopman operator—a linear but infinite-dimensional operator that describes the evolution of observables. We approximate this operator using Dynamic Mode Decomposition with Rank-Revealing QR (DMD_RRR).
Negative Sobolev Loss Functions
To improve robustness against noise, we move beyond Mean Squared Error. We propose loss functions based on negative-index Sobolev norms ($H^{-s}$). By using Fourier transforms to weight errors, these norms prioritize large-scale features (low frequency) and reduce sensitivity to small-scale discrepancies (high-frequency noise), effectively acting as an integrated low-pass filter.
Monitoring Training Dynamics
The eigenvalues of the Koopman operator serve as a diagnostic tool for the training state. Early in the process, eigenvalues clustered near zero indicate rapid learning and significant weight adjustments. As training stabilizes and approaches a fixed point, these eigenvalues migrate toward the unit circle.
Experimental Results
The methodology was evaluated using the MNIST dataset for spectral analysis and synthetic multi-scale signals for noise robustness testing.
Convergence Monitoring
On MNIST, the Koopman spectrum successfully tracked the transition from rapid learning to stabilization. By epoch 500, the eigenvalues clearly clustered around the unit circle, confirming that the system was nearing convergence.
Noise Robustness
In multi-scale signal reconstruction, standard $L^2$ loss failed as noise increased, capturing only the average of the noisy signal. In contrast, the "Sobolev loss 1" approach, which applies layer-wise scaling, maintained high reconstruction accuracy even when the noise magnitude equaled the signal magnitude ($\epsilon = 1.0$).
Conclusion
Koopman Mode Analysis provides a powerful framework for real-time monitoring of neural network training. While the Sobolev loss functions offer superior noise rejection, future research is needed to scale these methods to modern large-scale architectures and structured real-world noise.
Summary updated