Deep neural networks (DNNs) are notoriously opaque. While we know they achieve remarkable performance, understanding why a specific image triggers a "cat" label instead of a "dog" label remains a significant challenge. Most interpretability methods rely on "peeking" at the network's gradients or internal weights—a white-box approach.
However, biological vision researchers have faced this "black box" problem for decades. Unable to easily map every connection in a living brain, they developed methods to probe a system's behavior using only input stimuli and output responses. This paper, presented at ICLR 2020, bridges the gap between biological and artificial vision by applying classical neuroscience techniques to DNNs. By treating neural networks as "black box" observers—much like a human subject in a psychology lab—the authors reveal the internal templates and biases that drive machine learning decisions.
Probing the Black Box: Methods from Neuroscience
The authors employ primary techniques borrowed from psychophysics and neurophysiology to analyze DNNs across standard benchmarks (MNIST, Fashion-MNIST, CIFAR-10, and ImageNet):
- Classification Images: The researchers fed millions of white noise images into a classifier. By averaging all the noise patterns that the network categorized as a specific class (e.g., "Digit 7"), they created a "bias map." This map represents the network's internal "prototype" or template for that class.
- Spike Triggered Averaging (STA): Instead of looking at the whole network, STA focuses on a single neuron. By averaging the stimuli that cause a specific neuron to "fire" (reach high activation), researchers can visualize that neuron's receptive field—essentially what specific feature (like a vertical edge) it is looking for.
- Micro-stimulation: Borrowing from brain stimulation studies, the authors artificially boosted the bias of specific layers during inference to see how it shifted the network's "psychometric function"—the mathematical relationship between signal strength and classification accuracy.
Key Insights: CNNs See Like Us
One of the most striking findings is that Convolutional Neural Networks (CNNs) develop human-interpretable templates for objects. For the MNIST dataset, the bias maps generated from a CNN look remarkably like actual handwritten digits to the human eye.
Interestingly, maps generated from Multi-Layer Perceptrons (MLPs) or Recurrent Neural Networks (RNNs) look like gibberish to humans. This suggests that the convolutional architecture itself encourages "human-like" visual templates.
Using these derived bias maps as simple templates (via dot product) achieved 25.8% accuracy on MNIST and 23.7% on CIFAR-10. While lower than the CNNs themselves, this is significantly higher than the 10% chance level, proving the maps capture meaningful diagnostic information. Furthermore, the filters derived via STA were nearly identical to the actual learned convolutional kernels, validating that white noise analysis is a mathematically sound way to "reverse-engineer" a network's filters.
Note on the "8" Bias: In MNIST, the network tends to classify pure white noise as the digit "8" more often than any other class. This is likely because "8" has the most complex structure that overlaps with parts of all other digits.
Black-Box Attacks and Adversarial Defense
Because these bias maps represent exactly what the network "wants" to see, they can be weaponized or used for defense.
- Black-Box Attacks: By adding a faint version of a "7" bias map to an image of a "1", an attacker can easily fool the network into changing its prediction. This requires absolutely no access to the network's internal gradients, making it a potent black-box attack.
- Adversarial Defense: Conversely, classification images can "unveil" adversarial patches. If a network is trained on poisoned data, the bias maps will clearly show the location and shape of the hidden trigger (e.g., a small x-shape in the corner), allowing researchers to detect compromised models.
Limitations and Future Directions
While powerful, White Noise Analysis has its limits. The method struggles with high-dimensional, unaligned data. On ImageNet, for example, 1 million noise samples were not enough to cover all 1,000 classes effectively, resulting in many "empty" classes where no noise triggered a prediction. However, the authors found that using "structured noise" (like Gabor wavelets) or generative models (VAEs) can produce recognizable bias maps with as few as 10,000 samples.
Additionally, STA and classification images provide a "first-order" (linear) approximation of the system. They do not fully capture the complex, non-linear interactions that happen in the deeper layers of a network.
Looking forward, the authors suggest applying these tools to study "racial fairness" in facial recognition or investigating more complex architectures like CapsuleNets and GANs, proving that sometimes the best way to understand artificial intelligence is to treat it a bit more like biological intelligence.
Summary updated