Learning with Interpretable Structure from RNN

Bo-Jian Hou, Zhi-Hua ZhouarXiv 2018 · OpenReview

TL;DR

This paper proposes learning a Finite State Automaton (FSA) from Recurrent Neural Networks (RNNs), particularly gated RNNs, to make their decision-making process more interpretable. By clustering RNN hidden states, they demonstrate that FSAs can act as a trustworthy, human-simulatable proxy for complex RNNs, offering insights into their internal workings and guiding future RNN design.

What Problem Does This Paper Address?

Deep learning models, especially Recurrent Neural Networks (RNNs) like LSTMs and GRUs, have achieved remarkable success in sequential data processing. However, their "black box" nature makes it difficult to understand why they make certain predictions. This lack of interpretability is a significant hurdle in safety-critical applications (e.g., healthcare, autonomous systems) where understanding the decision-making process is crucial for trust and reliability. The paper specifically targets gated RNNs, whose internal mechanisms are even less understood than simpler RNNs. The goal is to bridge this interpretability gap by finding a simpler, more understandable structure that accurately reflects the RNN's behavior.

Methods

The core idea is to leverage the inherent structure within RNNs to learn a Finite State Automaton (FSA), which is known for its interpretability. The proposed method, LISOR (Learning with Interpretable Structure from gated RNN), involves three main steps:

  1. Train RNN Model: Standard RNN models (SRN, MGU, GRU, LSTM) are trained on the given data.
  2. Cluster Hidden States: The hidden states generated by the trained RNNs are collected. The paper proposes two clustering methods:
  3. k-means++: A standard clustering algorithm for grouping similar hidden states.
  4. k-means-x: An extension that incorporates "extra position features" to account for the temporal proximity of hidden states within a sequence, assuming states closer in time are also closer in space. Each cluster is treated as a state in the FSA.

  5. Learn Structured FSA: Based on the clustered hidden states, the FSA's components (alphabet, states, start state, accepting states, and transitions) are defined. The transitions are learned by observing how hidden states move between clusters as new input is processed. To ensure a deterministic FSA for easier interpretation, less frequent transitions are pruned.

Datasets

The paper uses both artificial and real-world datasets:

  • Artificial Tasks:
    • Task "0110": Identifying sequences containing "0110" within length-4 binary sequences. This is a simple task with 16 distinct cases.
  • Task "000": Determining if a sequence contains three consecutive zeros. This is a more complex task with an infinite instance space.
  • Real-World Task:
    • Sentiment Analysis: Using the IMDB dataset for movie review sentiment classification (positive/negative). For this task, word embeddings (word2vec) are used to represent words as numerical vectors.

Key Insights and Ideas

  • Hidden States Cluster: Gated RNNs, like their non-gated counterparts, exhibit a natural tendency for their hidden states to form clusters, suggesting an underlying structured behavior.
  • FSA as an Interpretable Proxy: Finite State Automata (FSAs) are significantly more interpretable than gated RNNs because their state transitions are directly observable and understandable by humans, without complex numerical computations.
  • Gate Effect: The number of gates in an RNN plays a role in its performance and the interpretability of its learned FSA. The paper suggests that while gates are important for preventing underfitting, having fewer gates (like in MGU) leads to more compact hidden state representations and thus simpler, more interpretable FSAs. "The less the better" seems to be a guiding principle.
  • Semantic Aggregation: The learned FSAs from text data reveal that states often represent aggregated semantic meanings, and transitions are driven by word classes with consistent sentiment (positive or negative).
  • Ensemble of FSAs: Combining multiple learned FSAs (from different initializations) through ensemble methods can improve classification performance, especially on more complex tasks.

Key Results

  • Interpretability Achieved: The LISOR method successfully learns interpretable FSAs from various gated RNNs (MGU, SRN, GRU, LSTM) that accurately reflect the RNNs' decision-making process.
  • FSA Trustworthiness: Experiments show that the learned FSAs are more "trustable" than the original RNNs, making them suitable for applications requiring high reliability.
  • MGU's Efficiency: The Minimal Gated Unit (MGU) consistently leads to FSAs with fewer clusters (states) for a given accuracy compared to GRU and LSTM, suggesting it's more efficient for learning interpretable structures.
  • Task Performance:
    • On the "0110" task, FSAs achieved 1.0 accuracy with a small number of clusters.
  • On the "000" task, FSAs achieved >0.7 accuracy, with MGU-based FSAs requiring fewer clusters.
  • On the sentiment analysis task, FSAs achieved competitive accuracy, with LISOR-x generally outperforming LISOR-k. Ensemble methods further boosted performance.
  • Consistency: Figure 6 and Table V demonstrate a strong consistency between the FSA's predictions and the original RNN's predictions, validating the FSA as a faithful representation.

Limitations and Future Work

  • FSA's Binary Classification Limitation: The current FSA formulation is best suited for binary classification tasks. The authors acknowledge this limitation and suggest exploring tools that can handle more complex tasks.
  • Accuracy vs. Interpretability Trade-off: While FSAs are more interpretable, they might not always reach the same peak accuracy as the original RNNs due to the simplification process (e.g., pruning less frequent transitions). However, the paper emphasizes that the goal is consistency and interpretability, not necessarily outperforming the RNN in raw accuracy.
  • Future Work:
    • Develop methods to handle more complex tasks beyond binary classification.
  • Incorporate the proposed FSA learning approach into abductive learning frameworks to bridge machine learning and logical reasoning.
  • Further explore the semantic interpretation of learned FSAs and their transitions.

Summary updated

Abstract

The interpretability of deep learning models has raised extended attention these years. It will be beneficial if we can learn an interpretable structure from deep learning models. In this paper, we focus on Recurrent Neural Networks~(RNNs) especially gated RNNs whose inner mechanism is still not clearly understood. We find that Finite State Automaton~(FSA) that processes sequential data has more interpretable inner mechanism according to the definition of interpretability and can be learned from RNNs as the interpretable structure. We propose two methods to learn FSA from RNN based on two different clustering methods. With the learned FSA and via experiments on artificial and real datasets, we find that FSA is more trustable than the RNN from which it learned, which gives FSA a chance to substitute RNNs in applications involving humans' lives or dangerous facilities. Besides, we analyze how the number of gates affects the performance of RNN. Our result suggests that gate in RNN is important but the less the better, which could be a guidance to design other RNNs. Finally, we observe that the FSA learned from RNN gives semantic aggregated states and its transition graph shows us a very interesting vision of how RNNs intrinsically handle text classification tasks.
BibTeX
@misc{hou2018learning,
  author = {Bo-Jian Hou and Zhi-Hua Zhou},
  title = {{Learning with Interpretable Structure from RNN}},
  year = {2018},
  eprint = {1810.10708},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1810.10708},
}
Figure