State Gradients for RNN Memory Analysis

Lyan Verwimp, Hugo Van hamme, Vincent Renkens, Patrick WambacqEMNLP 2018

TL;DR

This paper introduces a novel framework using state gradients to analyze what information Recurrent Neural Networks (RNNs), specifically LSTMs, retain from their input embeddings and for how long. By decomposing these gradients, the authors can quantify how well different word classes and specific linguistic properties are encoded in the network's memory, offering a more interpretable view into these complex models.

What Problem Does This Paper Address?

Neural networks, particularly deep learning models like RNNs, are incredibly powerful but often function as "black boxes." It's challenging to understand precisely what they learn and how they store information. While we can examine network weights, this doesn't directly reveal how specific input features are represented or retained over time within the network's internal states. This lack of interpretability hinders our ability to trust, debug, and improve these models. This paper aims to bridge this gap by providing a method to analyze the memory of RNNs, specifically focusing on how input information is transferred and preserved in the hidden state over sequential processing.

Methods

The core of the paper's contribution lies in its novel framework for analyzing RNN memory. Inspired by backpropagation, the authors compute the gradients of the RNN's state with respect to its input embeddings. Instead of using these gradients to update weights (as in training), they analyze the gradients themselves to understand information flow.

Key methodological steps include:

  • State Gradients: For a given input embedding at time step t-T and a state at time step t, they compute the gradient matrix G_t,T. This matrix captures how changes in the input embedding influence the state.
  • Delay Analysis: By varying the delay T, they can investigate how long information from an input is retained. A delay of T=0 relates to the current input, while larger T values look at past inputs.
  • Singular Value Decomposition (SVD): The gradient matrix G (averaged over time and/or specific word classes) is decomposed using SVD. The singular values (σ) and vectors (U, V) reveal:
  • V: Directions in the input embedding space that are best preserved in the RNN's state.
  • Σ: The magnitude of preservation for these directions.
  • U: Corresponding directions in the hidden state space.
  • Average Memory: By analyzing the largest singular values (σ₁) and the sum of all singular values (Σσ) across different delays T, they can quantify the overall memory retention of the RNN.
  • Tracking Specific Properties: To assess how well specific linguistic properties (like singular vs. plural nouns) are remembered, they define a property vector d (e.g., the difference between averaged embeddings of singular and plural nouns). They then compare this vector d with the dominant directions in V (obtained from SVD) using cosine similarity or by calculating a "relative memory" score.
  • Linear Separability Check: Before analyzing properties, they use logistic regression to check if the classes defining a property (e.g., singular vs. plural nouns) are linearly separable in the embedding space, ensuring the property is well-defined.

Datasets

  • Penn Treebank (PTB): Used for training LSTM language models. The paper notes using a normalized version ("PTB-norm") to better align with other datasets.
  • Wall Street Journal (WSJ): Used for pre-training word embeddings. The paper mentions using the CSR LM-1 corpus (LDC) for this, which is a much larger dataset (110M words) than PTB. Embeddings pre-trained on WSJ were found to yield better results in some analyses.

Key Insights and Ideas

  • RNNs Have Selective Memory: LSTMs don't just store everything equally. Information is retained, but the network becomes more "selective" over time, meaning the most important directions in the embedding space are preserved more strongly relative to others as the delay increases.
  • Pre-trained Embeddings Can Be a Double-Edged Sword: While pre-training embeddings on a larger corpus like WSJ can improve performance on certain tasks (like logistic regression for POS tagging), it can also lead to a more selective and potentially less informative memory in the LSTM for long-term dependencies, as indicated by lower absolute singular values. This suggests pre-trained embeddings might lack certain long-range relevant information.
  • Linguistic Properties are Remembered Differentially: The framework can reveal that certain word classes (like pronouns and nouns) have a stronger influence on the RNN's state than others (like adverbs or adjectives). Similarly, specific linguistic distinctions (like singular vs. plural) are remembered to varying degrees.
  • The "Black Box" is Becoming More Transparent: The state gradient approach offers a principled way to peek inside the RNN's memory, moving beyond intuition-based explanations to quantitative analysis of information retention.

Key Results

  • Memory Decay: For an LSTM trained on PTB-norm, a significant portion of input information is forgotten within about 5 time steps, but some information persists for over 20 time steps.
  • Selectivity Increase: The ratio of the largest singular value to the sum of all singular values increases with delay, indicating that the memory becomes more selective over time.
  • Class-Specific Memory: Pronouns and nouns have a stronger impact on the cell state compared to adjectives and adverbs in the jointly trained LM.
  • Property Retention: The singular-plural distinction for nouns was slightly better remembered than the common-proper noun distinction in the jointly trained LM.
  • Pre-trained Embeddings Impact: LMs using WSJ-pre-trained embeddings showed smaller absolute singular values and a higher degree of selectivity over time compared to LMs with jointly trained embeddings.
  • Logistic Regression Accuracy: Using WSJ-norm embeddings for logistic regression tasks (like POS tagging) generally yielded higher accuracies than PTB-norm embeddings, especially when the LM was also trained on WSJ embeddings.

Limitations and Future Work

  • Limited Scope of Analysis: The paper primarily focuses on the cell state of LSTMs. The authors suggest extending the analysis to hidden states/outputs of LSTMs and other types of RNNs.
  • Hyperparameter Sensitivity: The analysis of specific properties could be further explored with more extensive comparisons of different values for n (number of top singular vectors) and larger delays. The influence of hyperparameters like hidden layer size is also an area for future investigation.
  • Dataset Size: The authors note that the PTB dataset, while standard, is relatively small for language modeling, which might influence the learned representations. They address this by using larger WSJ embeddings.
  • Future Directions: The authors propose extending the research to track different properties, perform more extensive comparisons, and examine more specific contexts. They also suggest investigating the influence of hyperparameters.

Summary updated

BibTeX
@inproceedings{verwimp2018state,
  author = {Lyan Verwimp and Hugo Van hamme and Vincent Renkens and Patrick Wambacq},
  title = {{State Gradients for RNN Memory Analysis}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
}
Figure