An Operator Theoretic Approach for Analyzing Sequence Neural Networks

Ilan Naiman, Omri Azencot

TL;DR

KANN (Koopman Analysis of Neural Networks) introduces a global, operator-theoretic framework to peer inside the "black box" of trained sequence neural networks. By approximating complex, nonlinear hidden state dynamics with a single linear matrix, KANN leverages eigendecomposition to extract highly interpretable semantic features—such as sentiment-tracking $n$-grams or physical landmarks in ECG signals.

What Problem Does This Paper Address?

Sequence models like RNNs, LSTMs, and Transformers achieve remarkable results across language and time-series tasks, but understanding how they arrive at their decisions remains a major challenge. Previous attempts to reverse-engineer these networks often rely on local dynamical systems theory, such as fixed-point analysis. While useful, fixed-point analysis only explains network behavior in the immediate neighborhood of specific attractor states. This local constraint can lead to inconsistent or incomplete explanations of global network behavior.

This paper addresses this gap by providing a global, data-driven approach to interpret sequence models across their entire latent space, independent of specific fixed points.

Methods

The authors introduce KANN (Koopman Analysis of Neural Networks), a method rooted in Koopman operator theory. The core insight of Koopman theory is that a finite-dimensional nonlinear dynamical system can be fully represented as an infinite-dimensional linear operator acting on a space of observable functions.

To make this practical for neural networks, KANN extracts the hidden states ($h_t$) of a trained network and computes a low-rank linear approximation of the global dynamics through three steps:

  1. Basis Projection: Hidden states are projected onto a lower-dimensional coordinate system using either a data-driven basis (like truncated SVD or PCA) or a data-agnostic basis (like the Fast Fourier Transform).
  2. Linear Mapping: A least-squares optimization problem finds a moderate-sized transition matrix ($C$) that maps the hidden state at time $t$ to time $t+1$ in the spectral domain.
  3. Eigendecomposition: Because matrix $C$ is linear, standard linear algebra tools (eigenvalues and eigenvectors) can be used to untangle the global dynamics into independent, interpretable components.

Datasets

The authors evaluated KANN across three distinct sequence modeling tasks:

  • IMDB Reviews Dataset: A standard binary sentiment classification dataset used to evaluate text sequence processing.
  • ECG5000: A subset of the MIT-BIH arrhythmia database containing 5,000 heartbeat signals (each of length 140), used to evaluate an LSTM autoencoder trained for anomaly detection.
  • Synthetic Copy Task: A classic benchmark designed to test the long-term memory retention capabilities of recurrent networks.

Key Insights and Ideas

  • Global Linearity in Latent Space: Remarkably, even though neural networks use highly nonlinear activation functions, their trajectories in the hidden state space are sufficiently linear to be faithfully captured by a single transition matrix.
  • Eigenvectors as Semantic Feature Detectors: The dominant eigenvectors of the Koopman matrix act as interpretable feature trackers. In sentiment analysis, specific eigenvectors isolate positive words, negative words, or contextual phrases (like "not bad").
Figure 1: We display reviews where each word is shaded based on P
  • Capturing Physical Landmarks: In ECG analysis, conjugate pairs of eigenvectors perfectly isolate specific physical landmarks of a heartbeat (such as local extrema), proving that the network naturally encodes key physiological features.
Figure 2: We show the median of reconstructions of normal beats when projected to each of the first four dominant conjugate pairs of Koopman eigenvectors. The medians (orange, green, red, brown) are plotted on top of the original signals and their reconstruction medians. The dashed black lines indicate important features of the signals which are well captured by the Koopman eigenvectors.
  • Demystifying Autoencoder Boundaries: By analyzing silhouette scores of the Koopman embedding, the authors discovered that the ECG autoencoder doesn't actually learn a distinct boundary between normal and anomalous beats in its latent space. Instead, it maps all signals close to the manifold of normal beats; anomalies are caught simply because the network fails to reconstruct their unique features.

Key Results

  • High-Fidelity Replication: KANN closely mirrors the original network's classifications, achieving >99% matching accuracy on the IMDB sentiment task and >97% agreement on ECG classification.
  • Low Reconstruction Error: The relative error ($e_{rel}$) of the hidden states predicted linearly by KANN was incredibly low: $0.095$ for sentiment analysis and $0.0056$ for ECG classification, proving that matrix $C$ truly mimics the underlying network.
  • Hierarchical $n$-gram Discovery: Without retraining the network, KANN successfully separated 1-grams, 2-grams, and 3-grams into distinct, hierarchically ordered subspaces of eigenvectors based purely on the text inputs.

Limitations and Future Work

The primary limitation of the current framework is that it is strictly post-hoc: it can only analyze networks that have already finished training. The authors note that a promising direction for future work is to explore the Koopman framework during the training process, potentially using it to regularize or guide network dynamics dynamically.

Summary updated

Abstract

Analyzing the inner mechanisms of deep neural networks is a fundamental task in machine learning. Existing work provides limited analysis or it depends on local theories, such as fixed-point analysis. In contrast, we propose to analyze trained neural networks using an operator theoretic approach which is rooted in Koopman theory, the Koopman Analysis of Neural Networks (KANN). Key to our method is the Koopman operator, which is a linear object that globally represents the dominant behavior of the network dynamics. The linearity of the Koopman operator facilitates analysis via its eigenvectors and eigenvalues. Our method reveals that the latter eigendecomposition holds semantic information related to the neural network inner workings. For instance, the eigenvectors highlight positive and negative n-grams in the sentiments analysis task; similarly, the eigenvectors capture the salient features of healthy heart beat signals in the ECG classification problem.
BibTeX
@inproceedings{naiman2023operator,
  author = {Ilan Naiman and Omri Azencot},
  title = {{An Operator Theoretic Approach for Analyzing Sequence Neural Networks}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  url = {https://openreview.net/forum?id=CWmOvUkD9P},
}

Loading related content...

Figure