An Operator Theoretic Approach for Analyzing Sequence Neural Networks

Ilan Naiman, Omri Azencot

Abstract

Deep neural networks for sequence processing, such as RNNs and LSTMs, are powerful but notoriously difficult to interpret. Traditional interpretability methods often rely on local fixed-point analysis, which fails to capture the global logic of the model. We introduce KANN (Koopman Analysis of Neural Networks), a framework that leverages Koopman operator theory to approximate the nonlinear dynamics of trained sequence models with a single linear matrix. Our results demonstrate that KANN can extract high-level semantic features—such as sentiment-carrying n-grams in text or salient peaks in physiological signals—while maintaining high fidelity to the original model's behavior.

Introduction

Sequence models like Recurrent Neural Networks (RNNs) and Long Short-Term Memory (LSTM) networks are discrete-time dynamical systems. Understanding how these models process information is a significant challenge in AI safety and interpretability. While researchers have used fixed-point analysis to study behavior near stable states, this approach is inherently local. It describes the network's reaction to specific inputs but lacks a global perspective on the model's underlying logic.

To address this, we propose KANN, which turns the "black box" of sequence models into interpretable linear algebra. By moving into a space of "observable" functions, we can represent complex nonlinear dynamics as a linear system, allowing for a comprehensive spectral analysis of the network's state space.

Methods: The KANN Framework

The core of KANN is Koopman Theory, which posits that any nonlinear dynamical system can be represented linearly in an infinite-dimensional space. In practice, KANN uses a data-driven approximation:

  1. State Extraction: Hidden states ($h_t$) are extracted from a trained model as it processes a dataset.
  2. Linear Approximation: A moderate-sized matrix ($C$) is computed to describe the transition $h_t \to h_{t+1}$ using a least-squares approach similar to Dynamic Mode Decomposition (DMD).
  3. Spectral Analysis: An eigendecomposition of $C$ allows the network's trajectory to be decomposed into a sum of its eigenvectors and eigenvalues.

This linear operator $C$ represents the global logic of the model, independent of any single input sample.

Experimental Results

Semantic Feature Extraction in Text

We evaluated KANN on the IMDB movie review dataset using a GRU-based model. We found that specific eigenvectors of the Koopman matrix act as semantic filters. For instance, certain modes respond exclusively to positive sentiment, while others track negative sentiment.

Crucially, KANN captures context beyond single words. The framework identifies complex n-grams, such as negations ("not bad") and intensifiers ("terribly wrong"), demonstrating that the linear operator captures the network's ability to process nuanced linguistic structures.

KANN identifies semantic context, such as negations ('not bad') and intensifiers ('terribly wrong'), through specific eigenvectors.

The KANN-based reconstruction achieved over 99% agreement with the original network's classification output, proving that the linear approximation preserves the model's decision-making logic.

Salient Feature Identification in ECG Signals

In the ECG5000 dataset, an LSTM-based autoencoder was analyzed to identify normal vs. anomalous heartbeats. KANN's dominant eigenvectors naturally aligned with the "local extrema"—the characteristic peaks and valleys—of a healthy heartbeat.

Dominant eigenvectors in the ECG task align with salient peaks and valleys of heartbeat signals, revealing the network's internal coordinate system.

The relative error of hidden states predicted by the linear KANN model was remarkably low (0.0056), confirming that a linear operator can effectively capture the essence of complex LSTM dynamics.

Conclusion

KANN provides a robust, global tool for interpreting sequence neural networks. By transforming nonlinear dynamics into a linear spectral problem, it reveals the semantic "logic" learned by the model. Future work will explore incorporating Koopman-based constraints during the training phase and extending the framework to non-recurrent architectures like Transformers.

Summary updated

Abstract

Analyzing the inner mechanisms of deep neural networks is a fundamental task in machine learning. Existing work provides limited analysis or it depends on local theories, such as fixed-point analysis. In contrast, we propose to analyze trained neural networks using an operator theoretic approach which is rooted in Koopman theory, the Koopman Analysis of Neural Networks (KANN). Key to our method is the Koopman operator, which is a linear object that globally represents the dominant behavior of the network dynamics. The linearity of the Koopman operator facilitates analysis via its eigenvectors and eigenvalues. Our method reveals that the latter eigendecomposition holds semantic information related to the neural network inner workings. For instance, the eigenvectors highlight positive and negative n-grams in the sentiments analysis task; similarly, the eigenvectors capture the salient features of healthy heart beat signals in the ECG classification problem.
BibTeX
@inproceedings{naiman2023operator,
  author = {Ilan Naiman and Omri Azencot},
  title = {{An Operator Theoretic Approach for Analyzing Sequence Neural Networks}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  url = {https://openreview.net/forum?id=CWmOvUkD9P},
}
Figure