TL;DR
Recent claims that untrained (random) neural networks model the primary visual cortex (V1) better than trained ones may be an artifact of specific analysis choices. By conducting a "multiverse analysis" across 36 different pipelines, this paper shows that the "untrained advantage" disappears or reverses when using different normalization methods or predictive readouts, suggesting that trained networks are actually superior when looking at the directional structure of neural responses.
What Problem Does This Paper Address?
In the quest to understand how the human brain processes vision, researchers often compare artificial neural networks (ANNs) to fMRI data. A standard tool for this is Representational Similarity Analysis (RSA), which compares the "similarity structure" of stimuli in a model versus the brain.
A surprising and recurring finding in recent years is that networks with random, untrained weights sometimes perform as well as—or even better than—trained networks at modeling the primary visual cortex (V1). This suggests that the brain's early visual representations might be a product of fixed architecture rather than learning.
However, RSA involves many "hidden" researcher choices: How do you normalize the data? Which distance metric (correlation, Euclidean, cosine) do you use? This paper asks a critical question: Is the "untrained advantage" a robust biological insight, or is it just a quirk of how we choose to analyze the data?
Methods
The author, Nils Leutenegger, re-examined his own previous findings using a more rigorous, multi-pronged approach:
- Multiverse Analysis: Instead of one "standard" pipeline, the author tested 36 different combinations of feature normalization (raw, centered, scaled, z-scored), distance measures, and voxel normalization.
- Ridge Readout: Moving beyond RSA, the author used Ridge Regression to see how well network features could actually predict voxel-wise brain activity. This is generally considered a more stringent test of a model's explanatory power.
- Component Decomposition: The author mathematically split the response patterns into two parts:
- Mean-profile component: How much a stimulus aligns with the average response of all units.
- Angular component: The specific "direction" or pattern of the response, independent of the mean.
- Baselines: The study included a simple "luminance-and-contrast bank" as a control to see if complex neural networks are actually doing anything more than capturing basic image statistics.
Datasets
- THINGS-fMRI: A high-quality dataset involving three subjects viewing 720 distinct object concepts.
- Models:
- CIFAR-trained CNNs: Custom networks trained with various rules (Backprop, Feedback Alignment, Predictive Coding, STDP) and their untrained counterparts.
- ImageNet Models: Six standard architectures (AlexNet, VGG11, ResNet18, ResNet50, ConvNeXt, ViT) in both trained and untrained states.
Key Insights and Ideas
- The "Mean-Profile" Trap: The perceived advantage of untrained networks in RSA was almost entirely driven by the "mean-profile" component—essentially, how much the units respond on average to a stimulus. This component correlates strongly with simple image luminance. When this was removed, the untrained advantage vanished.
- Readout Matters: When switching from RSA to a fitted Ridge Readout, the results flipped. Trained networks consistently outperformed untrained ones at every resolution and pooling grid.
- The Angular Advantage: When looking specifically at the angular component (the "shape" of the representation), trained networks were ahead in both V1 and the higher-level Lateral Occipital Complex (LOC).
- Baselines are Hard to Beat: Surprisingly, a simple bank of luminance and contrast filters predicted V1 activity better than any of the neural networks tested. This suggests that much of what we measure in V1 with fMRI might just be low-level image statistics that don't require a deep network to explain.
Key Results
- Ridge Regression: Untrained networks scored lower than trained ones (median correlation difference of -0.043).
- Multiverse Flip: Across the 36 analysis variations, the "untrained advantage" was only present in a minority of cases (5 of 28 distinct analyses). In 9 cases, the trained network was significantly better, and in 14, there was no clear winner.
- RSA Covariance: Raw correlation-distance RSA scores were almost perfectly predicted (Spearman 0.97) by how well the model aligned with the mean-profile of the voxels.
- LOC Consistency: In the higher-level visual area (LOC), trained networks were superior across almost all analysis choices.
Limitations and Future Work
- Small Sample Size: The study relies on only three subjects from a single dataset. While the data is high-quality, the results describe these specific subjects and may not generalize to the entire population.
- Low Absolute Performance: Even the "best" models only explain a tiny fraction of the variance in the fMRI signal (median $r \approx 0.086$).
- Biological vs. Measurement Artifact: The paper cannot determine if the "mean-profile" component represents a real neural mechanism (like contrast normalization) or is simply an artifact of how fMRI (the BOLD signal) measures brain activity.
- Future Direction: The author recommends that future "model-brain" papers should always report sensitivity to centering, include simple stimulus-statistic baselines, and report norm and angular components separately to avoid being misled by analysis artifacts.
Summary updated