Neural networks are powerful but notoriously opaque, often criticized as "black boxes." While they excel at pattern recognition, understanding how they arrive at a solution remains a significant challenge. This opacity is particularly pronounced in Cascade-Correlation (CC) networks. Unlike standard feed-forward networks with neat, uniform layers, CC networks are generative—they dynamically build their topology during training by adding hidden units one by one and creating "cross-connections" that bypass layers entirely.
Standard analytical tools, which usually look at either raw weights or hidden unit activations in isolation, fail here because information in a CC network is highly distributed across these irregular connections. To address this, researchers Thomas Shultz and Jeffrey Elman extended a technique called "contribution analysis" to decode the inner workings of these complex models, revealing how they exploit symmetries and functional categories to solve non-linear problems.
Methodology: Adapting Contribution Analysis
To visualize the logic of these messy networks, the authors adapted Contribution Analysis, originally proposed by Sanger (1989). They define a "contribution" as the triple product of:
- The weight of a connection.
- The activation of the sending unit.
- The sign of the output target (to measure "correctness").
The formula is expressed as $c_{ijk} = w_{jk} \cdot a_{ij} \cdot 2t_{ki}$. By including the target's sign, the authors can distinguish between connections that push the network toward the right answer and those that inhibit it.
Instead of Sanger’s complex 3D slicing, they create a 2D matrix of contributions versus input patterns. They then apply Principal Components Analysis (PCA) with Varimax rotation to this matrix. This identifies the primary "dimensions" or axes of the network's decision-making process, making the distributed representations interpretable.
Experiments and Key Insights
The method was tested on three benchmark tasks of increasing complexity, revealing fascinating insights into how the networks structured their internal logic.
Continuous XOR
In a 2D expansion of the classic exclusive-OR problem (with 100 input pairs creating four quadrants), PCA successfully reduced the network's representation to two components. These two components perfectly clustered the 100 input patterns into their four respective quadrants, demonstrating the method's ability to extract clean logical boundaries.
Comparative Arithmetic
In a custom task, the network had to determine if the sum or product of two numbers was greater than, less than, or equal to a third comparison value. The analysis showed that hidden units do not simply "do addition"; rather, they specialize in specific logical sub-tasks, such as distinguishing "equality" from "inequality," while broader "greater than/less than" distinctions are handled by other components.
For addition, two PCA components were sufficient to separate the three possible answers ($>$, $<$, $=$). Multiplication required 3–4 components, reflecting the higher non-linearity of the task. When trained on both operations, the network developed a specific component dedicated to distinguishing between the "adding" and "multiplying" contexts.
The Two-Spirals Problem
The two-spirals problem is a famously difficult non-linear discrimination task requiring the network to distinguish between two interlocking spirals that wrap around an origin three times.
The PCA revealed that the network exploits the inherent 180-degree symmetry of the task. If the network learns that a point $(x, y)$ belongs to Spiral A, the PCA shows it simultaneously "understands" that $(-x, -y)$ belongs to Spiral B. This suggests a level of systematicity in neural networks that some cognitive scientists previously thought was impossible. The network didn't just memorize points; it partitioned the coordinate space into symmetrical, "hourglass-shaped" sectors of responsibility.
Limitations and Future Directions
While powerful, the approach has limitations. The authors noted that scaling contributions by the sign of the output target might occasionally obscure the full picture of the network's internal representation, as the network itself doesn't "see" the target during inference.
However, the potential applications are highly promising. The authors suggest this technique could be used to create a "developmental trace"—analyzing a network at different stages of training to see if it progresses through human-like stages of learning. Furthermore, PCA could be used for lesion prediction: identifying which hidden units are critical for specific functions, allowing researchers to predict exactly how removing a specific unit would degrade the network's overall performance.
Summary updated