Understanding how neural networks represent knowledge is notoriously difficult. This paper demonstrates that applying Principal Components Analysis (PCA) to the variance-covariance matrix of "contributions" (the product of unit activations and connection weights) provides a much clearer picture of a network's logic than standard correlation-based methods. By refusing to standardize the data, the authors preserve the critical information stored in the magnitude of connection weights.
The Problem with Standard PCA in Neural Networks
Neural networks are often criticized as "black boxes." While we can see the weights and the activations, it is hard to tell how a network actually solves a problem, especially in "cross-connected" architectures like Cascade-Correlation where connections bypass layers.
Standard analysis techniques usually fall into two traps:
- Weight Analysis: Looking at raw weights is overwhelming because they are numerous and vary wildly between training runs.
- Activation Analysis: Looking only at hidden unit activations ignores the "strength" of the connections (the weights) that actually deliver that information to the next layer.
Previous researchers used PCA on "contributions" to solve this, but they typically used a correlation matrix. This standardizes the data, effectively forcing every connection to have the same variance. The authors argue this is a mistake: it ignores the fact that some weights are much more important (have higher variance) than others.
A Refined Method: Unstandardized Contribution Analysis
The authors propose a refined version of Contribution Analysis that respects the scale of the network's learned parameters:
- The Contribution: Defined as the product of a sending unit's activation and the weight of the connection to the output.
- Unstandardized & Unscaled: Unlike previous methods, they do not scale contributions by the "target" (which the network wouldn't know during a forward pass anyway) and they do not standardize the variables.
- Variance-Covariance PCA: They apply PCA to the variance-covariance matrix. This ensures that connections with high-variance weights—the ones doing the "heavy lifting"—dominate the primary components.
- Interpretation: They use Varimax rotation to make the resulting components easier to interpret and plot "component scores" to see how different input patterns map onto the network's internal logic.
Key Insights and Ideas
By evaluating three classic connectionist benchmarks (Continuous XOR, Comparative Arithmetic, and the Two-Spirals Problem), the study revealed several core insights:
- Weights are Information: Standardizing data treats all variables as equally important. In a neural net, the variation in weight sizes is a primary way the network encodes knowledge. By using the variance-covariance matrix, the authors keep this "weight information" intact.
- Task vs. Input Separation: The PCA successfully separated the "essential" task (e.g., is the answer greater or less than?) from "incidental" features (e.g., what were the raw input values?).
- Realism: By using unscaled contributions, the analysis reflects the network's actual state during the feed-forward (inference) phase, rather than relying on "cheat" information from the training targets.
Key Results
The unstandardized PCA approach yielded highly interpretable results across the benchmark tasks:
- Continuous XOR: Three components accounted for 96% of the variance. Component 1 (44.3%) specifically handled the core task of distinguishing positive from negative output targets.
- Arithmetic: Four components accounted for 88.9% of the variance. Component 1 (51.1%) separated "greater than" from "less than" answers, while Component 2 (20.2%) clearly distinguished addition from multiplication operations.
- The "Aha" Moment for Two-Spirals: Previous attempts to analyze the famously difficult Two-Spirals problem using PCA failed to show how the network actually distinguished one spiral from the other. This unstandardized approach finally revealed the specific components the network uses to pull the two shapes apart. Six components accounted for 97.9% of the variance. Crucially, Components 2 and 4 together provided a clear linear separation of the two spirals, misclassifying only 11.3% of the points.
Limitations and Future Work
- Architecture Specificity: The study focused on Cascade-Correlation networks. While the logic should hold for other architectures, the paper does not test standard deep MLP or CNN structures.
- Scaling to Modern Nets: The networks used here had 5 to 16 hidden units. It remains to be seen how this method handles modern "wide" or "deep" models with thousands of hidden units.
- Predictive Lesioning: The authors suggest that these PCA results could be used to predict the effects of "lesioning" (deleting) specific units. If a unit has a high loading on a specific component, removing it should theoretically impair the specific function served by that component.
Summary updated