TL;DR
Understanding how recurrent and sequence neural networks make decisions has long relied on local fixed-point approximations. This paper introduces the Koopman Analysis of Neural Networks (KANN), a data-driven method that fits a global linear operator to the latent dynamics of trained sequence models. By analyzing the eigenvectors and eigenvalues of this operator, researchers can decompose complex black-box representations into interpretable features, such as sentiment-tracking $n$-grams and salient physiological landmarks in time series.
What Problem Does This Paper Address?
Despite the widespread success of recurrent models (RNNs, LSTMs, GRUs) and attention architectures on sequential data, explaining their internal decision mechanisms remains difficult. Prior dynamical systems-based interpretability techniques analyze the network's latent space near fixed points. However, fixed-point analysis is inherently local—it describes behavior only in the immediate neighborhood of an attractor state and struggles to explain trajectories that travel further away in state space.
The authors propose a global, data-driven alternative rooted in Koopman operator theory to describe sequence dynamics across the entire latent trajectory using linear algebra.
Methods
The paper introduces KANN (Koopman Analysis of Neural Networks), a white-box framework that requires access only to the model's hidden states $H \in \mathbb{R}^{s \times n \times k}$ (batch size $s$, sequence length $n$, hidden dimension $k$):
- Spectral Representation: The hidden states are projected onto an orthonormal basis $B$ (derived via truncated SVD of the hidden states): $\tilde{H}_\tau = H_\tau B$.
- Linear Operator Estimation: A transition matrix $C$ is fit to map spectral coefficients across consecutive time steps using least-squares minimization: $$\min_{C} \sum_{t=1}^{n-1} \|\tilde{H}_t C - \tilde{H}_{t+1}\|_F^2$$
- Spectral Decomposition: An eigendecomposition $C = V \Lambda U$ is performed (where $U = V^{-1}$). The rows of $U$ form the Koopman eigenvectors, creating a natural coordinate system where dynamics decouple into independent, scalar-multiplied modes.
- Memory Horizons: Eigenvalues $\lambda_j$ define the temporal persistence of each eigenvector mode. For $|\lambda_j| < 1$, the memory horizon is explicitly calculated as $\tau_j(\epsilon) = \frac{\log(\epsilon)}{\log(|\lambda_j|)}$, representing how many steps a feature persists before decaying by threshold $\epsilon$.
Datasets
The authors evaluate KANN on two benchmark domains (and a synthetic task in the appendix):
- IMDB Reviews Dataset: Binary sentiment classification (25,000 train / 25,000 test reviews) modeled with a 128-dimensional embedding and a 256-unit GRU layer.
- ECG5000: A 5,000-sample subset of the MIT-BIH Arrhythmia Database containing heartbeat time series (sequence length 140; ~60% normal, ~40% anomalous). Evaluated using an LSTM autoencoder ($F_{\text{enc}}$ and $F_{\text{dec}}$, hidden size 64) trained only on normal heartbeats.
- The Copy Problem (Appendix F): A synthetic memory task used to benchmark retention against prior fixed-point baselines.
Key Insights and Ideas
- Higher-Dimensional Attractors in Sentiment Analysis: Previous dynamical studies concluded that sentiment RNNs operate primarily via a 1D "line attractor" separating positive from negative sentiment. KANN reveals that the attracting manifold is higher-dimensional:
- Specific conjugate eigenvector pairs independently track positive words ($u_1, u_2$) versus negative words ($u_3, u_4$), with eigenvalues $|\lambda| > 0.99$ yielding memory horizons $\tau > 800$ tokens.
- Higher-index eigenvectors isolate contextual phrases and modifiers. For instance, eigenvector $u_8$ specifically activates on negation and amplifier bigrams such as "not bad" and "terribly wrong".
- Uncovering How Anomaly Detectors Actually Work: In the ECG LSTM autoencoder, projecting hidden states onto dominant conjugate pairs reveals that individual Koopman modes isolate specific physiological landmarks of the heartbeat (e.g., the initial trough at $t=3$, the broad wave between $t \in [35, 75]$, the primary peak at $t=103$, and the secondary peak at $t=133$).
- Latent Prototype Learning: Silhouette score analysis showed that while reconstructed signals cleanly separate normal and anomalous beats, the Koopman latent space does not maintain a sharp boundary. Instead, the network projects all beats onto the normal manifold; anomalous beats simply fail to reconstruct well, exposing anomalies via residual reconstruction error.
Key Results
- Faithful Dynamics Reproduction: Linear propagation via matrix $C$ reproduced the true network outputs with $> 99\%$ agreement on IMDB sentiment classification and $> 97\%$ agreement on ECG anomaly detection.
- Low Latent Trajectory Error: The average relative error of predicted hidden states ($e_{\text{rel}}$) remained low:
- $0.095$ for sentiment analysis (batch size 64).
- $0.0056$ for ECG reconstruction (batch size 145).
Limitations and Future Work
- Lossy Finite Approximation: True Koopman operators are infinite-dimensional; compressing the dynamics into a moderate-sized matrix $C$ necessarily discards some higher-order nonlinear phenomena.
- Post-Hoc Analysis: KANN currently analyzes static, fully trained networks. The authors highlight extending this framework to track how dynamical modes evolve during training as the primary direction for future research.
Summary updated