TL;DR
This paper introduces a novel "enmeshment loss" to train neural networks that are inherently more modular. By encouraging disjoint clusters of neurons, the authors demonstrate that these modular networks lead to significantly more interpretable circuits and reduced computational complexity for understanding individual class predictions.
What Problem Does This Paper Address?
Interpreting the inner workings of deep neural networks is a major challenge, especially as models grow larger and more complex. While methods like circuit discovery aim to map specific functionalities to subnetworks, the sheer scale and interconnectedness of these circuits often make them difficult to analyze. The authors propose that splitting a network into more modular, independently understandable clusters could be a key to unlocking better interpretability. However, existing pre-trained networks are found to be highly "unclusterable," meaning their components are deeply intertwined. This paper tackles the problem of how to train networks to be modular from the ground up, making them amenable to this cluster-based interpretability approach.
Methods
The core innovation is the "enmeshment loss", a regularization term added to the standard cross-entropy loss during training. This loss function penalizes weights that connect neurons between clusters, thereby encouraging the formation of non-interacting, modular clusters.
To identify these clusters, the authors adapt Bipartite Spectral Graph Clustering (BSGC). This method uses the network's weight matrix (or accumulated gradients) to construct a similarity graph between input and output neurons of a layer. Spectral decomposition and k-means clustering are then applied to partition the neurons into k distinct bipartite clusters.
The effectiveness of their approach is evaluated using two key metrics:
- Class-wise accuracy: This measures how well the model predicts a specific class when individual clusters are activated or deactivated, indicating specialization.
- Effective Circuit Size (ECS): Using automated pruning techniques (like ACDC), the authors estimate the size of the subnetwork (circuit) responsible for predicting each class. A smaller ECS implies a more interpretable and potentially simpler circuit.
Datasets
The authors train and evaluate their methods on two standard datasets:
- MNIST: Used for training Multi-Layer Perceptrons (MLPs).
- CIFAR-10: Used for training Convolutional Neural Networks (CNNs).
Key Insights and Ideas
- Pre-trained networks are inherently unclusterable: The paper demonstrates that standard training methods result in highly entangled neuron clusters, hindering modular interpretability.
- Enmeshment loss is key to modularity: Introducing the "enmeshment loss" during training effectively forces the network to organize into more distinct, non-interacting clusters.
- Modularity leads to specialized circuits: The resulting clusters show a tendency to specialize in recognizing specific classes, as evidenced by class-wise accuracy analysis.
- Modularity simplifies circuit discovery: By creating more disjoint clusters, the effective circuit size required to explain class predictions is significantly reduced, making analysis more tractable.
Key Results
- The proposed "enmeshment loss" successfully increases clusterability, achieving scores as high as 0.99 on CIFAR-10.
- For CIFAR-10, the clustered models show improved class-wise accuracy when individual clusters are activated, suggesting specialization.
- The Effective Circuit Size (ECS) for predicting individual labels is significantly reduced in clustered models compared to un-clustered ones. On average, un-clustered models have 61.25% more parameters in their effective circuits for CIFAR-10, and 61.25% more parameters for MNIST. This reduction in ECS directly translates to easier interpretability.
- The authors show that even with a moderate number of clusters (e.g., k=4), significant improvements in clusterability are observed.
Limitations and Future Work
- Scalability: The authors acknowledge that the Pareto frontier between clusterability and performance might become challenging as they scale to harder tasks and larger models.
- Language Models: They express interest in applying these modularity insights to train and align large language models, hoping to improve interpretability and control over unwanted behaviors.
- Broader Interpretability Goals: Future work could explore clustering other network components, such as attention heads or using linear probes, to further advance mechanistic interpretability.
- Dataset Specificity: While tested on MNIST and CIFAR-10, the generalizability to more complex datasets and architectures remains an open question.
Summary updated