Knowledge-Driven Interpretation of Convolutional Neural Networks

Riccardo Massidda, Davide Bacciu

TL;DR

This paper introduces a novel framework for interpreting Convolutional Neural Networks (CNNs) by aligning their internal representations with human-understandable concepts from an ontology like WordNet. It goes beyond single neuron analysis to identify "neural circuits" and demonstrates their importance for classification tasks, offering a more holistic view of network behavior.

What Problem Does This Paper Address?

Deep learning models, especially CNNs, are increasingly used in critical applications, but their "black box" nature makes them difficult to understand and trust. Existing interpretation methods often focus on individual neurons or simple directions, neglecting the distributed nature of representations and the rich semantic relationships between concepts. This leaves a gap in understanding how complex, interconnected concepts are represented and utilized within a network.

Methods

The core of the paper is a "semantic alignment framework" that connects neural network components (specifically, directions within layers) to concepts from a structured ontology (WordNet). Key methodological contributions include:

  • Ontology-Driven Alignment: The framework leverages a semantic ontology (like WordNet) to align concepts with neural directions. This allows for "mask propagation," where if a network component is aligned with a specific concept (e.g., "dog"), it can also be implicitly aligned with its parent concepts (e.g., "canine," "animal") without direct annotation.
  • Neural Directions: Instead of just individual neurons, the framework considers "neural directions" (linear combinations of neuron activations) as the units of interpretation. This better captures the distributed nature of representations.
  • Alignment Measures: Several measures are proposed to quantify the alignment between a neural direction and a concept, including Jaccard similarity (IoU), F1 score, and a novel recall-based measure ($\sigma_\epsilon$) that is more robust to polysemantic neurons (neurons responding to multiple concepts).
  • Neural Circuits: The framework identifies "neural circuits" by clustering semantically aligned directions that are also architecturally connected within the network. This is achieved by building a directed graph of aligned pairs and extracting connected components.
  • Direction Learning: The paper also proposes a method to learn new neural directions that are semantically aligned with a given concept, rather than just analyzing existing ones.

Datasets

  • Broden Dataset: Used for training and evaluation. The authors extended the original Broden dataset by associating its labels with WordNet synsets, creating a more ontologically rich resource.
  • ImageNet: Used for supplementary experiments, with concept masks generated from bounding-box annotations.
  • Places-365 Dataset: Used for evaluating CNNs trained on scene classification tasks.

Key Insights and Ideas

  • Beyond Single Neurons: Focusing on "neural directions" and "neural circuits" provides a more nuanced and accurate understanding of network representations than analyzing individual neurons in isolation.
  • The Power of Ontologies: Explicitly incorporating structured knowledge (like WordNet) significantly enhances interpretability by enabling concept propagation and revealing richer semantic associations.
  • Emergence of Neural Circuits: The research demonstrates that CNNs naturally form semantically rich and architecturally connected "neural circuits" that are crucial for their performance.
  • Polysemanticity is Key: The proposed $\sigma_\epsilon$ measure is better suited for interpreting polysemantic neurons, which are common in real-world networks, by focusing on recall rather than strict precision.
  • Circuits are Pivotal: Ablating these identified neural circuits leads to a significant drop in classification accuracy, highlighting their functional importance.

Key Results

  • The proposed mask propagation strategy, leveraging ontological relationships, significantly increased the number of alignable concepts compared to methods relying solely on direct annotations.
  • The $\sigma_\epsilon$ measure consistently identified a higher number of aligned concepts and meaningful neural circuits compared to IoU, especially for polysemantic neurons.
  • Experiments on popular CNN architectures (AlexNet, ResNet, DenseNet) showed the emergence of semantically related neural circuits.
  • Ablation studies demonstrated that removing entire neural circuits led to a substantial drop in classification accuracy for related classes, while ablating individual units within a circuit had a less pronounced effect. For example, in AlexNet, ablating a circuit related to clothing concepts caused a significant accuracy drop for clothing-related classes.

Limitations and Future Work

  • Approximation of Receptive Fields: The paper acknowledges that the current method approximates the relationship between neural directions and their receptive fields by scaling masks. Future work could address the effects of striding and padding in convolutional units.
  • Computational Cost: Weight estimation for edges within neural circuits is computationally expensive, leading to analysis being limited to these circuits.
  • Future Directions: The authors suggest further research into the influence of directions on other network outcomes and exploring the relationship between concepts within circuits. They also release their "Bisturi" library and an extended Broden dataset to facilitate future research.

Summary updated

Abstract

Since the widespread adoption of deep learning solutions in critical environments, the interpretation of artificial neural networks has become a significant issue. To this end, numerous approaches currently try to align human-level concepts with the activation patterns of artificial neurons. Nonetheless, they often understate two related aspects: the distributed nature of neural representations and the semantic relations between concepts. We explicitly tackled this interrelatedness by defining a novel semantic alignment framework to align distributed activation patterns and structured knowledge. In particular, we detailed a solution to assign to both neurons and their linear combinations one or more concepts from the WordNet semantic network. Acknowledging semantic links also enabled the clustering of neurons into semantically rich and meaningful neural circuits. Our empirical analysis of popular convolutional networks for image classification found evidence of the emergence of such neural circuits. Finally, we discovered neurons in neural circuits to be pivotal for the network to perform effectively on semantically related tasks. We also contribute by releasing the code that implements our alignment framework.
BibTeX
@inproceedings{massidda2022knowledge,
  author = {Riccardo Massidda and Davide Bacciu},
  title = {{Knowledge-Driven Interpretation of Convolutional Neural Networks}},
  booktitle = {ECML/PKDD},
  year = {2022},
  doi = {10.1007/978-3-031-26387-3\_22},
}
Figure