TL;DR
This paper introduces a novel framework for interpreting Convolutional Neural Networks (CNNs) by aligning their internal representations with human-understandable concepts from an ontology like WordNet. It goes beyond single neuron analysis to identify "neural circuits" and demonstrates their importance for classification tasks, offering a more holistic view of network behavior.
What Problem Does This Paper Address?
Deep learning models, especially CNNs, are increasingly used in critical applications, but their "black box" nature makes them difficult to understand and trust. Existing interpretation methods often focus on individual neurons or simple directions, neglecting the distributed nature of representations and the rich semantic relationships between concepts. This leaves a gap in understanding how complex, interconnected concepts are represented and utilized within a network.
Methods
The core of the paper is a "semantic alignment framework" that connects neural network components (specifically, directions within layers) to concepts from a structured ontology (WordNet). Key methodological contributions include:
- Ontology-Driven Alignment: The framework leverages a semantic ontology (like WordNet) to align concepts with neural directions. This allows for "mask propagation," where if a network component is aligned with a specific concept (e.g., "dog"), it can also be implicitly aligned with its parent concepts (e.g., "canine," "animal") without direct annotation.
- Neural Directions: Instead of just individual neurons, the framework considers "neural directions" (linear combinations of neuron activations) as the units of interpretation. This better captures the distributed nature of representations.
- Alignment Measures: Several measures are proposed to quantify the alignment between a neural direction and a concept, including Jaccard similarity (IoU), F1 score, and a novel recall-based measure ($\sigma_\epsilon$) that is more robust to polysemantic neurons (neurons responding to multiple concepts).
- Neural Circuits: The framework identifies "neural circuits" by clustering semantically aligned directions that are also architecturally connected within the network. This is achieved by building a directed graph of aligned pairs and extracting connected components.
- Direction Learning: The paper also proposes a method to learn new neural directions that are semantically aligned with a given concept, rather than just analyzing existing ones.
Datasets
- Broden Dataset: Used for training and evaluation. The authors extended the original Broden dataset by associating its labels with WordNet synsets, creating a more ontologically rich resource.
- ImageNet: Used for supplementary experiments, with concept masks generated from bounding-box annotations.
- Places-365 Dataset: Used for evaluating CNNs trained on scene classification tasks.
Key Insights and Ideas
- Beyond Single Neurons: Focusing on "neural directions" and "neural circuits" provides a more nuanced and accurate understanding of network representations than analyzing individual neurons in isolation.
- The Power of Ontologies: Explicitly incorporating structured knowledge (like WordNet) significantly enhances interpretability by enabling concept propagation and revealing richer semantic associations.
- Emergence of Neural Circuits: The research demonstrates that CNNs naturally form semantically rich and architecturally connected "neural circuits" that are crucial for their performance.
- Polysemanticity is Key: The proposed $\sigma_\epsilon$ measure is better suited for interpreting polysemantic neurons, which are common in real-world networks, by focusing on recall rather than strict precision.
- Circuits are Pivotal: Ablating these identified neural circuits leads to a significant drop in classification accuracy, highlighting their functional importance.
Key Results
- The proposed mask propagation strategy, leveraging ontological relationships, significantly increased the number of alignable concepts compared to methods relying solely on direct annotations.
- The $\sigma_\epsilon$ measure consistently identified a higher number of aligned concepts and meaningful neural circuits compared to IoU, especially for polysemantic neurons.
- Experiments on popular CNN architectures (AlexNet, ResNet, DenseNet) showed the emergence of semantically related neural circuits.
- Ablation studies demonstrated that removing entire neural circuits led to a substantial drop in classification accuracy for related classes, while ablating individual units within a circuit had a less pronounced effect. For example, in AlexNet, ablating a circuit related to clothing concepts caused a significant accuracy drop for clothing-related classes.
Limitations and Future Work
- Approximation of Receptive Fields: The paper acknowledges that the current method approximates the relationship between neural directions and their receptive fields by scaling masks. Future work could address the effects of striding and padding in convolutional units.
- Computational Cost: Weight estimation for edges within neural circuits is computationally expensive, leading to analysis being limited to these circuits.
- Future Directions: The authors suggest further research into the influence of directions on other network outcomes and exploring the relationship between concepts within circuits. They also release their "Bisturi" library and an extended Broden dataset to facilitate future research.
Summary updated