A Discussion about Explainable Inference on Sequential Data via Memory-Tracking

Biagio La Rosa, Roberto Capobianco, Daniele NardiDP@AI*IA 2021 · OpenReview

Introduction

Deep learning models are often criticized as "black boxes"—we see what they decide, but we rarely understand why. This problem is even more acute when dealing with sequential data (like text or time-series), where models must remember past information to make future decisions. In this work, a method is proposed to peer inside the "brain" of Memory Augmented Neural Networks (MANNs). By tracking how a model writes to and reads from its external memory, we can pinpoint exactly which parts of an input sequence triggered a specific output.

Methodology

The core of the approach is a two-part system: a "Simplified Differentiable Neural Computer" (DNC) and an accompanying explanation module that tracks memory access patterns.

Simplified Differentiable Neural Computer (Simplified DNC)

The original DNC architecture was modified to enhance interpretability:

  • Allocation-only Writing: Unlike standard DNCs that use content-based addressing to overwrite or update cells, this version prioritizes writing to empty cells. This creates a near one-to-one mapping between an input step and a memory location.
  • Reduced Recurrence: Feedback loops at the memory level were removed to prevent "input dispersion," ensuring that the information retrieved from memory remains tied to its original source.

The Explanation Module

This module records where information was written (mapping input $x_t$ to memory cell $i$) and what was read (tracking cells with the highest "read weightings"). It ranks inputs based on the frequency and intensity of their retrieval. If the model reads cell $i$ repeatedly before making a prediction, the input associated with cell $i$ is flagged as the primary explanation.

Visualizing memory-tracking explanations in the Story Cloze Test.

Experimental Evaluation

The method was evaluated on two distinct types of tasks:

  1. Modified T-maze: A discrete control task where an agent moves through a corridor and must remember a specific symbol seen earlier to decide whether to turn North or South at a junction.
  2. Story Cloze Test: A commonsense reasoning task where the model is given a four-sentence premise and must choose the correct fifth sentence to end the story.

Key Results

The validity of the explanations was tested by feeding the model only the "best" premise (as ranked by the module). If the model could still predict the right answer with just that one sentence, the explanation was deemed a "prime implicant"—the core reason for the logic.

Average explanation accuracy across different premise selection strategies.

As shown in the table above, when the model was fed only the "Best" premise, it achieved 73.3% accuracy on the development set. In contrast, feeding it the "Worst" premise dropped accuracy to 53.3%, which is near random. This confirms the module successfully identified the load-bearing parts of the story. In the T-maze task, the "matching accuracy" (how often the model read the correct relevant corridor step) perfectly tracked with the model's learning curve, showing that as the agent improved, it increasingly relied on the correct stored hint.

Discussion and Limitations

While promising, the module relies on hyperparameters like $N_c$ (the number of top read cells to track), which can introduce noise if set incorrectly. Additionally, MANNs can be unstable to train and may occasionally ignore external memory in favor of internal states. Future research aims to extend these concepts to other domains, such as image classification, where memory can be used to track which parts of an image most influence a specific category prediction.

Summary updated

BibTeX
@inproceedings{rosa2021discussion,
  author = {Biagio La Rosa and Roberto Capobianco and Daniele Nardi},
  title = {{A Discussion about Explainable Inference on Sequential Data via Memory-Tracking}},
  booktitle = {DP@AI*IA},
  year = {2021},
  url = {https://openreview.net/forum?id=SAy3l4ZG2S},
}
Figure