TL;DR
KANN (Koopman Analysis of Neural Networks) introduces a global, operator-theoretic framework to peer inside the "black box" of trained sequence neural networks. By approximating complex, nonlinear hidden state dynamics with a single linear matrix, KANN leverages eigendecomposition to extract highly interpretable semantic features—such as sentiment-tracking $n$-grams or physical landmarks in ECG signals.
What Problem Does This Paper Address?
Sequence models like RNNs, LSTMs, and Transformers achieve remarkable results across language and time-series tasks, but understanding how they arrive at their decisions remains a major challenge. Previous attempts to reverse-engineer these networks often rely on local dynamical systems theory, such as fixed-point analysis. While useful, fixed-point analysis only explains network behavior in the immediate neighborhood of specific attractor states. This local constraint can lead to inconsistent or incomplete explanations of global network behavior.
This paper addresses this gap by providing a global, data-driven approach to interpret sequence models across their entire latent space, independent of specific fixed points.
Methods
The authors introduce KANN (Koopman Analysis of Neural Networks), a method rooted in Koopman operator theory. The core insight of Koopman theory is that a finite-dimensional nonlinear dynamical system can be fully represented as an infinite-dimensional linear operator acting on a space of observable functions.
To make this practical for neural networks, KANN extracts the hidden states ($h_t$) of a trained network and computes a low-rank linear approximation of the global dynamics through three steps:
- Basis Projection: Hidden states are projected onto a lower-dimensional coordinate system using either a data-driven basis (like truncated SVD or PCA) or a data-agnostic basis (like the Fast Fourier Transform).
- Linear Mapping: A least-squares optimization problem finds a moderate-sized transition matrix ($C$) that maps the hidden state at time $t$ to time $t+1$ in the spectral domain.
- Eigendecomposition: Because matrix $C$ is linear, standard linear algebra tools (eigenvalues and eigenvectors) can be used to untangle the global dynamics into independent, interpretable components.
Datasets
The authors evaluated KANN across three distinct sequence modeling tasks:
- IMDB Reviews Dataset: A standard binary sentiment classification dataset used to evaluate text sequence processing.
- ECG5000: A subset of the MIT-BIH arrhythmia database containing 5,000 heartbeat signals (each of length 140), used to evaluate an LSTM autoencoder trained for anomaly detection.
- Synthetic Copy Task: A classic benchmark designed to test the long-term memory retention capabilities of recurrent networks.
Key Insights and Ideas
- Global Linearity in Latent Space: Remarkably, even though neural networks use highly nonlinear activation functions, their trajectories in the hidden state space are sufficiently linear to be faithfully captured by a single transition matrix.
- Eigenvectors as Semantic Feature Detectors: The dominant eigenvectors of the Koopman matrix act as interpretable feature trackers. In sentiment analysis, specific eigenvectors isolate positive words, negative words, or contextual phrases (like "not bad").
- Capturing Physical Landmarks: In ECG analysis, conjugate pairs of eigenvectors perfectly isolate specific physical landmarks of a heartbeat (such as local extrema), proving that the network naturally encodes key physiological features.
- Demystifying Autoencoder Boundaries: By analyzing silhouette scores of the Koopman embedding, the authors discovered that the ECG autoencoder doesn't actually learn a distinct boundary between normal and anomalous beats in its latent space. Instead, it maps all signals close to the manifold of normal beats; anomalies are caught simply because the network fails to reconstruct their unique features.
Key Results
- High-Fidelity Replication: KANN closely mirrors the original network's classifications, achieving >99% matching accuracy on the IMDB sentiment task and >97% agreement on ECG classification.
- Low Reconstruction Error: The relative error ($e_{rel}$) of the hidden states predicted linearly by KANN was incredibly low: $0.095$ for sentiment analysis and $0.0056$ for ECG classification, proving that matrix $C$ truly mimics the underlying network.
- Hierarchical $n$-gram Discovery: Without retraining the network, KANN successfully separated 1-grams, 2-grams, and 3-grams into distinct, hierarchically ordered subspaces of eigenvectors based purely on the text inputs.
Limitations and Future Work
The primary limitation of the current framework is that it is strictly post-hoc: it can only analyze networks that have already finished training. The authors note that a promising direction for future work is to explore the Koopman framework during the training process, potentially using it to regularize or guide network dynamics dynamically.
Summary updated