Neural Networks are Surprisingly Modular

Daniel Filan, Shlomi Hod, Cody Wild, Andrew Critch, Stuart RussellarXiv 2020 · OpenReview

TL;DR

This paper introduces a novel way to measure "modularity" in neural networks by treating their weights as a graph. They discover that training and pruning neural networks, especially with dropout, significantly increases this modularity compared to random networks. This suggests that learning processes might inherently favor structured, modular representations, potentially paving the way for more interpretable AI.

What Problem Does This Paper Address?

Neural networks, particularly deep ones, are often seen as "black boxes." While they achieve impressive performance, understanding how they arrive at their decisions is challenging due to the seemingly inscrutable nature of their learned weights. The learned weights are typically treated as a dense, complex web, making it difficult to pinpoint specific functions or components within the network. This lack of interpretability hinders debugging, trust, and the ability to engineer more robust and understandable AI systems. This paper aims to address this by developing a method to quantify and investigate the internal structure of neural networks, specifically their modularity.

Methods

The core of this paper is the introduction of a quantifiable measure of modularity for Multi-Layer Perceptrons (MLPs). They conceptualize the neural network as a weighted, undirected graph where neurons are vertices and connections are edges weighted by the absolute value of the connection's weight. Modularity is then defined using concepts from graph clustering, specifically the "normalized cut" (n-cut) metric. A low n-cut indicates that neurons within a partition (a "module") are strongly connected internally but weakly connected to neurons outside the partition.

To measure this, they employ spectral clustering, a standard technique for partitioning graphs. They use the normalized Laplacian of the graph and its eigenvectors to embed neurons into a lower-dimensional space, then apply k-means clustering. The n-cut of the resulting clustering serves as their modularity score. They then compare the n-cuts of trained networks against those of randomly shuffled networks (preserving sparsity but not structure) to assess if training and pruning genuinely promote modularity.

Datasets

The authors trained and evaluated their models on the following standard image datasets:

  • MNIST: A dataset of handwritten digits.
  • Fashion-MNIST: A dataset of clothing items.
  • CIFAR-10: A dataset of small color images across 10 classes.

For CIFAR-10, they down-sampled images to 28x28 pixels and converted them to grayscale to match the format of MNIST and Fashion-MNIST. They also created "unlearnable" random datasets for specific experiments to disentangle the effects of learning from accuracy.

Key Insights and Ideas

  • Learning Induces Modularity: The most striking finding is that the process of training neural networks, especially when combined with weight pruning, leads to significantly more modular network structures than random initialization. This suggests that learning algorithms might inherently favor or discover modular representations.
  • Dropout Amplifies Modularity: Surprisingly, training with dropout not only doesn't harm modularity but actively increases it, often substantially. This contradicts the intuition that dropout, by forcing neurons to rely on multiple inputs, might lead to less specialized structures.
  • Pruning is Key to Modularity: While training itself shows some increase in modularity, pruning appears to be a crucial step in revealing and enhancing this modular structure.
  • Modularity vs. Accuracy: The paper suggests that modularity is not directly tied to accuracy. Networks can be highly modular but have moderate accuracy (e.g., on CIFAR-10), and vice-versa. This decouples the structural property from task performance.
  • Functional Significance of Modules: Exploratory experiments (lesioning sub-modules) indicate that these identified modules can have functional importance, with some sub-modules being critical for network performance.

Key Results

  • Trained and pruned MLPs consistently showed lower n-cuts (higher modularity) than random networks with similar sparsity. For instance, on Fashion-MNIST, 100% of pruned networks had lower n-cuts than random shuffles.
  • Networks trained with dropout exhibited significantly lower n-cuts than those trained without dropout across all tested datasets. The distributions of n-cuts for dropout-trained networks showed essentially no overlap with those trained without.
  • Experiments on "unlearnable" random datasets indicated that modularity increase is not solely a byproduct of the optimizer (SGD) but is linked to the network learning to represent information, even if that information is random.
  • Lesioning experiments revealed that certain identified sub-modules were "important," meaning their removal caused a statistically significant drop in accuracy (over 1 percentage point).

Limitations and Future Work

  • Network Size and Complexity: The study primarily focuses on small MLPs trained on small image datasets. The authors acknowledge that it's unclear if these findings extend to larger, more complex architectures like Convolutional Neural Networks (CNNs) or Recurrent Neural Networks (RNNs).
  • Definition of Modularity: While they introduce a graph-theoretic definition, the paper notes that "modularity" has other technical definitions in network science, and their approach is specific to the learned weights.
  • Understanding "Why": The paper raises questions about why networks become modular and why dropout has this effect. These are identified as areas for future research.
  • Functional Interpretation: While they show some modules are important, a deeper understanding of what specific functions these modules perform and how they interact remains an open challenge.
  • Direct Regularization: The authors pose the question of whether modularity can be directly regularized for during training, which could be a promising avenue for designing more interpretable networks.

Summary updated

Abstract

The learned weights of a neural network are often considered devoid of scrutable internal structure. To discern structure in these weights, we introduce a measurable notion of modularity for multi-layer perceptrons (MLPs), and investigate the modular structure of MLPs trained on datasets of small images. Our notion of modularity comes from the graph clustering literature: a "module" is a set of neurons with strong internal connectivity but weak external connectivity. We find that training and weight pruning produces MLPs that are more modular than randomly initialized ones, and often significantly more modular than random MLPs with the same (sparse) distribution of weights. Interestingly, they are much more modular when trained with dropout. We also present exploratory analyses of the importance of different modules for performance and how modules depend on each other. Understanding the modular structure of neural networks, when such structure exists, will hopefully render their inner workings more interpretable to engineers. Note that this paper has been superceded by "Clusterability in Neural Networks", arxiv:2103.03386 and "Quantifying Local Specialization in Deep Neural Networks", arxiv:2110.08058!
BibTeX
@misc{filan2020neural,
  author = {Daniel Filan and Shlomi Hod and Cody Wild and Andrew Critch and Stuart Russell},
  title = {{Neural Networks are Surprisingly Modular}},
  year = {2020},
  eprint = {2003.04881},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2003.04881},
}
Figure