Response Characterization for Auditing Cell Dynamics in Long Short-term Memory Networks

Ramin M. Hasani, Alexander Amini, Mathias Lechner, Felix Naser, Radu Grosu, Daniela Rus

Abstract

While Long Short-term Memory (LSTM) networks have achieved remarkable success in sequential data processing, their internal dynamics remain difficult to interpret. This paper proposes a novel "testbench" for LSTMs that treats individual cells as dynamical systems. By subjecting trained cells to fundamental step and sinusoidal signals, the researchers can quantify neuron-level characteristics such as settling time and response magnitude. This framework provides a rigorous, mathematically grounded method to audit the "black box" of recurrent networks, enabling the identification of critical neurons and the detection of overfitting signatures.

Introduction

Recurrent Neural Networks (RNNs) and LSTMs are notoriously difficult to interpret, which poses a significant challenge for their deployment in safety-critical fields like autonomous driving or medical diagnostics. Traditional interpretability methods often rely on subjective feature visualizations or computationally expensive gate-ablation. There is a clear need for a robust framework to analyze how individual hidden states contribute to global network behavior across diverse data types, from financial time-series to biological sequences.

Methods: The Response Characterization Pipeline

The authors introduce a systematic pipeline that isolates individual LSTM cells from a trained network and exposes them to two baseline inputs derived from classical control theory:

  1. Step Response: By applying a sudden jump in input, researchers measure Settling Time (to distinguish "fast" vs. "slow" neurons) and the Delta Response (to indicate a cell's influence or "strength").
  2. Sinusoidal Response: Cyclic inputs are used to measure Amplitude and Phase/Frequency, further quantifying how a cell aligns with or shifts away from the input signal.
The Response Characterization Pipeline treats LSTM cells as dynamical systems to extract insights and validate them via ablation.

These metrics allow for a comprehensive ranking of every cell in the network. To validate these rankings, the authors perform Ablation Analysis, systematically "knocking out" neurons to confirm if those predicted to be important by the characterization metrics actually cause the largest drop in model performance.

Experimental Setup

The method was evaluated on four diverse sequential datasets to demonstrate generalizability:

  • Sequential MNIST: Hand-written digits processed pixel-by-pixel.
  • S&P 500 Stock Prices: Financial time-series regression.
  • CO2 Concentrations: Environmental data from the Mauna Loa volcano.
  • Protein Sequencing: Classification of secondary protein structures.

Key Insights and Results

Functional Specialization

The settling time metric successfully identified a functional divide between "fast-reacting" neurons, which handle immediate changes, and "slow" neurons, which act as long-term integrators of information. As shown in the Sequential MNIST results, cells exhibit a wide distribution of these dynamic properties.

Characterization of LSTM cells on Sequential MNIST, showing cell outputs and the distribution of metrics like settling time, delta response, and sine amplitude.

Predicting Importance and Scaling

The "Delta Response" and "Sine Amplitude" metrics proved to be excellent predictors of cell importance. In ablation tests, neurons with high values in these metrics caused the most significant spikes in error when removed. Furthermore, the authors observed a $1/N$ Scaling Law: as network capacity ($N$) increases, the relative contribution of each individual cell decreases proportionally, quantifying how larger networks distribute knowledge more thinly.

Overfitting Signatures

In the Protein-Net dataset, a rapid decay in settling time was observed as network capacity increased. The authors identified this as a signature of overfitting, where the network shifts from learning general dynamics to memorizing specific training samples.

Conclusion

This research provides a lightweight, scalable framework for auditing LSTM dynamics. By bridging the gap between deep learning and control theory, it offers both a "personality" profile for individual neurons and a global view of network behavior. Future work may involve applying this testbench during the training phase to observe how these dynamics emerge and investigating more complex input signals to capture the full dynamic space of nonlinear recurrent systems.

Summary updated

Abstract

In this paper, we introduce a novel method to interpret recurrent neural networks (RNNs), particularly long short-term memory networks (LSTMs) at the cellular level. We propose a systematic pipeline for interpreting individual hidden state dynamics within the network using response characterization methods. The ranked contribution of individual cells to the network's output is computed by analyzing a set of interpretable metrics of their decoupled step and sinusoidal responses. As a result, our method is able to uniquely identify neurons with insightful dynamics, quantify relationships between dynamical properties and test accuracy through ablation analysis, and interpret the impact of network capacity on a network's dynamical distribution. Finally, we demonstrate generalizability and scalability of our method by evaluating a series of different benchmark sequential datasets.
BibTeX
@inproceedings{hasani2018response,
  author = {Ramin M. Hasani and Alexander Amini and Mathias Lechner and Felix Naser and Radu Grosu and Daniela Rus},
  title = {{Response Characterization for Auditing Cell Dynamics in Long Short-term Memory Networks}},
  booktitle = {IJCNN},
  year = {2018},
  doi = {10.1109/ijcnn.2019.8851954},
}
Figure