Abstract
Deep neural networks for sequence processing, such as RNNs and LSTMs, are powerful but notoriously difficult to interpret. Traditional interpretability methods often rely on local fixed-point analysis, which fails to capture the global logic of the model. We introduce KANN (Koopman Analysis of Neural Networks), a framework that leverages Koopman operator theory to approximate the nonlinear dynamics of trained sequence models with a single linear matrix. Our results demonstrate that KANN can extract high-level semantic features—such as sentiment-carrying n-grams in text or salient peaks in physiological signals—while maintaining high fidelity to the original model's behavior.
Introduction
Sequence models like Recurrent Neural Networks (RNNs) and Long Short-Term Memory (LSTM) networks are discrete-time dynamical systems. Understanding how these models process information is a significant challenge in AI safety and interpretability. While researchers have used fixed-point analysis to study behavior near stable states, this approach is inherently local. It describes the network's reaction to specific inputs but lacks a global perspective on the model's underlying logic.
To address this, we propose KANN, which turns the "black box" of sequence models into interpretable linear algebra. By moving into a space of "observable" functions, we can represent complex nonlinear dynamics as a linear system, allowing for a comprehensive spectral analysis of the network's state space.
Methods: The KANN Framework
The core of KANN is Koopman Theory, which posits that any nonlinear dynamical system can be represented linearly in an infinite-dimensional space. In practice, KANN uses a data-driven approximation:
- State Extraction: Hidden states ($h_t$) are extracted from a trained model as it processes a dataset.
- Linear Approximation: A moderate-sized matrix ($C$) is computed to describe the transition $h_t \to h_{t+1}$ using a least-squares approach similar to Dynamic Mode Decomposition (DMD).
- Spectral Analysis: An eigendecomposition of $C$ allows the network's trajectory to be decomposed into a sum of its eigenvectors and eigenvalues.
This linear operator $C$ represents the global logic of the model, independent of any single input sample.
Experimental Results
Semantic Feature Extraction in Text
We evaluated KANN on the IMDB movie review dataset using a GRU-based model. We found that specific eigenvectors of the Koopman matrix act as semantic filters. For instance, certain modes respond exclusively to positive sentiment, while others track negative sentiment.
Crucially, KANN captures context beyond single words. The framework identifies complex n-grams, such as negations ("not bad") and intensifiers ("terribly wrong"), demonstrating that the linear operator captures the network's ability to process nuanced linguistic structures.
The KANN-based reconstruction achieved over 99% agreement with the original network's classification output, proving that the linear approximation preserves the model's decision-making logic.
Salient Feature Identification in ECG Signals
In the ECG5000 dataset, an LSTM-based autoencoder was analyzed to identify normal vs. anomalous heartbeats. KANN's dominant eigenvectors naturally aligned with the "local extrema"—the characteristic peaks and valleys—of a healthy heartbeat.
The relative error of hidden states predicted by the linear KANN model was remarkably low (0.0056), confirming that a linear operator can effectively capture the essence of complex LSTM dynamics.
Conclusion
KANN provides a robust, global tool for interpreting sequence neural networks. By transforming nonlinear dynamics into a linear spectral problem, it reveals the semantic "logic" learned by the model. Future work will explore incorporating Koopman-based constraints during the training phase and extending the framework to non-recurrent architectures like Transformers.
Summary updated