Abstract
We introduce a new approach to understanding trained sequence neural models: the Koopman Analysis of Neural Networks (KANN) method. Motivated by the relation between time-series models and self-maps, we compute approximate Koopman operators that encode well the latent dynamics. Unlike other existing methods whose applicability is limited, our framework is global, and it has only weak constraints over the inputs. Moreover, the Koopman operator is linear, and it is related to a rich mathematical theory. Thus, we can use tools and insights from linear analysis and Koopman Theory in our study. For instance, we show that the operator eigendecomposition is instrumental in exploring the dominant features of the network. Our results extend across tasks and architectures as we demonstrate for the copy problem, and ECG classification and sentiment analysis tasks.
Introduction
Deep neural networks, particularly sequence models like Recurrent Neural Networks (RNNs), Long Short-Term Memory (LSTM) networks, and Gated Recurrent Units (GRUs), are powerful tools for modeling temporal data but are often criticized as "black boxes." Understanding how these models process information over time is a significant challenge. Existing interpretability methods, such as fixed-point analysis, typically focus on local behavior near attractor states. This approach can lead to inconsistent results and fails to explain the network's global logic when it is far from those points.
We propose KANN (Koopman Analysis of Neural Networks), a global framework that uses Koopman theory to linearize the latent dynamics of sequence models. By approximating the nonlinear hidden state transitions with a single linear operator, we can leverage the power of spectral analysis to extract semantic features and decision-making logic across the entire latent space.
Methodology
The KANN procedure transforms the complex dynamics of a neural network into a mathematically rigorous linear framework through four key steps:
- State Extraction: Hidden states ($h_t$) are collected as the trained model processes input sequences.
- Basis Projection: These high-dimensional states are projected onto a lower-dimensional subspace, typically using truncated Singular Value Decomposition (SVD).
- Linear Approximation: A least-squares problem is solved to find a matrix ($C$) that best approximates the transition $h_t \to h_{t+1}$ linearly.
- Spectral Analysis: An eigendecomposition of the matrix $C$ identifies Koopman eigenvectors, representing the fundamental directions of the system's dynamics.
Experimental Evaluation
We evaluated KANN across three distinct domains to demonstrate its versatility:
- Sentiment Analysis (IMDB): Identifying positive and negative linguistic features.
- Physiological Signal Classification (ECG5000): Detecting salient features in heartbeats from the MIT-BIH arrhythmia database.
- Memory Benchmarks (Copy Task): Testing long-term state propagation and reproduction accuracy.
Analysis of Latent Dynamics
A central insight of this work is that the latent dynamics of sequence models are "sufficiently linear" to be captured by a Koopman operator. This linearity allows us to use eigenvectors to identify semantic roles within the network. In sentiment analysis, specific eigenvectors were found to track positive or negative sentiment, as well as contextual modifiers like negations.
Furthermore, in the ECG classification task, the dominant Koopman modes naturally aligned with clinically significant features, such as the QRS complex. This indicates that the model's internal representations are grounded in the physical reality of the signals it processes.
Results and Fidelity
The linear Koopman operator achieved high fidelity, showing over 99% classification agreement with the original nonlinear network on the sentiment task and over 97% on the ECG task. In the Copy Task, KANN significantly outperformed local fixed-point baselines (RENN) in predicting future hidden states, maintaining over 80% accuracy in reproducing digits. These results were consistent across RNN, LSTM, and GRU architectures, suggesting that KANN is a robust, architecture-agnostic tool for model interpretation.
Conclusion and Future Work
KANN provides a powerful, global perspective on the inner workings of sequence models. While it requires white-box access to hidden states and relies on low-rank approximations, it provides a level of interpretability that local methods cannot match. Future research could explore integrating Koopman-based loss functions during training to encourage the development of more stable and naturally interpretable latent dynamics.
Summary updated