Improving Interpretability of Deep Neural Networks with Semantic Information

Yinpeng Dong, Hang Su, Jun Zhu, Bo Zhang

TL;DR

This research introduces a method to open the "black box" of Deep Neural Networks (DNNs) by aligning internal neurons with human-understandable semantic topics derived from text captions. By incorporating an "interpretive loss" during training, the researchers developed a video captioning model that is not only more accurate but also allows human users to diagnose and correct errors through a simple feedback loop.

What Problem Does This Paper Address?

While DNNs achieve state-of-the-art performance in computer vision, they are notoriously difficult to interpret. It is often unclear why a model predicts one action over another, such as "dancing" versus "walking." This lack of transparency hinders trust in critical fields like medicine or autonomous driving and makes debugging an inefficient process of trial and error.

Comparison between an opaque black-box system and the proposed interpretation system.

Existing interpretability methods often focus on simpler architectures or attempt to visualize neuron activations post-hoc. This paper addresses these gaps by integrating interpretability directly into the learning process for complex CNN-RNN architectures and providing a mechanism for non-expert humans to interact with and refine the model.

Methods

The authors propose a framework called LSTM-I (Interpretive LSTM) specifically for video captioning. The approach consists of four primary components:

  1. Topic Extraction: Latent Dirichlet Allocation (LDA) is used to extract 100 semantically meaningful topics (e.g., "dog," "dance," "road") from a corpus of human-provided video descriptions.
  2. Interpretive Loss: A novel "interpretive loss" term is added to the standard objective function during training. This forces the hidden features generated by the video encoder to be predictive of the topics present in the video's captions, effectively regularizing neurons to align with human concepts.
  3. Prediction Difference Maximization: To map specific neurons to topics, an algorithm measures the drop in topic prediction when a particular neuron is deactivated. The neuron causing the largest drop is identified as the representative for that topic.
  4. Human-in-the-Loop Learning: If the model fails, a human can provide a "missing" topic. The system then boosts the activation of corresponding neurons and fine-tunes the encoder to improve future performance on similar videos.
The LSTM-I architecture featuring joint training with interpretive and task losses.

Key Insights and Ideas

  • Interpretability Improves Performance: Contrary to the belief that interpretability requires a performance trade-off, the authors found that interpretive loss actually improved captioning accuracy. Forcing neurons to learn distinct semantic concepts helped the model capture superior temporal information.
  • Neuron Specialization: In standard models, activations are often noisy. In the LSTM-I model, activations became "peaky," meaning specific neurons became highly specialized experts for specific topics, such as a "horse" or "woman" neuron.
  • Generalization through Semantics: Features aligned with human semantics are more general and less task-specific. This was demonstrated when interpretable features outperformed standard features in transfer learning for action recognition.

Key Results

  • Video Captioning: The LSTM-I model achieved a BLEU score of 0.446, significantly outperforming the baseline LSTM-B (0.416).
  • Action Recognition: When transferred to the UCF11 dataset, the interpretable model achieved 91.13% accuracy, surpassing both the baseline (88.50%) and a model trained from scratch (90.13%).
  • Qualitative Success: The human-in-the-loop procedure successfully refined 17 out of 20 failed predictions. For instance, providing a missing "polar bear" topic allowed the model to correctly identify the animal in subsequent frames.

Limitations and Future Work

While promising, the current approach uses a simplified mapping of one topic per neuron, whereas artificial and biological neurons often exhibit "polysemy" (responding to multiple related concepts). Additionally, the human-in-the-loop process currently requires manual input, and the method's effectiveness depends on the availability of a rich text corpus for topic extraction. Future research could explore automated correction suggestions and applications in data-scarce domains.

Summary updated

BibTeX
@inproceedings{dong2017improving,
  author = {Yinpeng Dong and Hang Su and Jun Zhu and Bo Zhang},
  title = {{Improving Interpretability of Deep Neural Networks with Semantic Information}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2017},
  url = {https://openreview.net/forum?id=HyEOFyz_-S},
}
Figure