Deep networks with two parallel visual pathways have become a leading account of why the brain splits vision into a "what" stream for object identity and a "where/how" stream for motion and action. Trained only to predict upcoming video frames, such a network splits on its own: one branch comes to resemble the mouse's ventral visual areas and the other its dorsal areas. Every test of this account so far has averaged neurons by brain area, treating each area as a uniform pool. Cortex is not uniform. Within any area, genetically distinct cell types, such as neurons that project down to the brainstem and colliculus versus neurons that talk to other cortical areas, carry different signals and play different causal roles. We propose to ask whether the model's two streams correspond to brain areas or to cell types. The test reuses the mouse recordings on which the area-level result was built, which were themselves collected in genetically labeled cell populations.
The question matters in both directions. If "dorsalness" in the model tracks a cell type across areas, then the area-level story is partly a sampling artifact, and dual-stream models gain a sharper, testable mapping onto physical wiring. If it tracks areas regardless of cell type, then cell-type diversity sits orthogonal to the stream computation, and that is also a useful constraint. Either answer resolves a tension the literature currently leaves open: are streams set by training objectives, by parallel architecture, or by the cell classes the cortex builds them from?
Core Claim
We claim that model-to-brain stream alignment can be broken down into an area component and a cell-type component. We further claim that a reliability-controlled analysis will show a nonzero cell-type component. Specifically, we expect subcortically projecting pyramidal-tract (PT) lines to prefer the model's dorsal branch more than intratelencephalic (IT) lines in the same area. This is the first test of dual-stream models at cell-type resolution. It differs from the closest work, Bakhtiari et al., in three ways: it does not pool neurons by area, it controls for the large differences in signal reliability between cell types, and it asks whether the cell-type mapping depends on the training objective.
Grounding in Prior Work
The motivating result is Bakhtiari et al.. They trained a two-pathway 3D convolutional network, ResNet-2p, with contrastive predictive coding (a self-supervised loss that rewards predicting future frames' features). They found by representational similarity analysis (RSA, comparing the pattern of stimulus-to-stimulus distances in model and brain) that one pathway matches VISp, VISlm and VISpm while the other matches VISal and VISam in the Allen Brain Observatory. They pooled all recorded cells by area, even though the dataset spans 13 Cre-driver lines. Choi et al. reached a similar area-level conclusion with human fMRI using separate "where" and "what" networks and separate objectives. The voxels there cannot resolve cell types at all, which is why we work in mouse.
The cell-type side supplies the hypothesis. Musall et al. imaged Fezf2 (PT), PlxnD1 (IT) and corticostriatal neurons cortex-wide. They found that IT activity has higher dimensionality, that PT neurons carry the dominant causal role in parietal cortex during an auditory decision task, and that cell type can be decoded from activity patterns alone. Yao et al. showed that PT and IT neurons split into 8 and 12 projection subtypes, with PT axons reaching the superior colliculus, an action-oriented target consistent with a "how" pathway. Neither study used visual stream tasks or pixel-level models. This proposal supplies both.
Two papers constrain the method. POYO-CAP showed that Allen Cre lines differ sharply in predictability, with predictable lines about 1.93x more informative per sample. Any naive per-line alignment score will therefore partly measure noise rather than computation, and our design controls for this explicitly. POYO+ showed that a neural decoder's latent space separates cell types at 67.93% accuracy. Cell types are therefore distinguishable from activity, but that model maps neural activity to behavior and says nothing about which visual computation each type carries. Dipoppa et al. showed that locomotion reconfigures Pvalb, Vip and Sst interactions in V1. This motivates a secondary, running-stratified analysis for interneuron lines.
Research Program
We first rebuild the Bakhtiari pipeline: train ResNet-2p with contrastive predictive coding on UCF101 and confirm the published dissociation. As a sanity anchor, the dorsal branch should retain roughly 40% accuracy on random-dot motion at 20% coherence, as they report.
We then extract Allen Brain Observatory responses to the natural movie stimuli. Recordings are organized by area, Cre line, imaging depth and session. The first concrete step is to inventory which of the 13 lines are PT-like, IT-like, or interneuron lines, and how they distribute across the six areas. The file does not list them, and the whole design depends on having at least one PT-like and one IT-like line recorded in both a ventral and a dorsal area.
For every cell in every (area, line, depth) group, we compute alignment to each model branch in two ways. The first is RSA, for comparability with Bakhtiari. The second is cross-validated ridge-regression predictivity, divided by that group's split-half noise ceiling. From these we form a stream preference index,
$$ S = \frac{A_d - A_v}{A_d + A_v}, $$
where $A_d$ and $A_v$ are the noise-ceiling-normalized alignments to the dorsal and ventral branches, taking each branch's best layer. Groups are subsampled to equal neuron counts.
We fit a mixed-effects model of $S$ with area, cell line, depth and their interaction as fixed effects and mouse as a random effect. The key quantity is the variance in $S$ explained by cell line beyond area and depth.
Finally, we repeat the analysis across model variants to probe the source of any effect. The variants are CPC ResNet-2p, ResNet-2p trained with supervised action recognition (which Bakhtiari found collapses both branches to ventral-like), single-pathway ResNet-1p, and a Choi-style two-objective network. Comparing them tells us whether the cell-type mapping comes from self-supervised prediction, from the two-branch architecture, or from separate objectives.
Baselines
The naive reference point is the area-only model. Each cell's stream preference is predicted entirely from its area label, which is exactly the assumption implicit in Bakhtiari's pooling. Cell line beats this baseline if adding it raises held-out explained variance in $S$ by a margin whose bootstrap interval over mice excludes zero. The comparison uses identical cells, folds and noise ceilings.
The strongest existing method is Bakhtiari's area-pooled RSA itself, run on our reproduced model with the same movies. The file reports no correlation value for it ("no reported result found on Lacuna for ResNet-2p RSA magnitude"). We therefore treat our reproduction as the baseline and report its numbers alongside ours. The behavioral anchor we can cite is ~40% random-dot accuracy at 20% coherence for its dorsal branch.
A third baseline is a hand-built motion-energy and Gabor feature model with no learning. Cell-type differences in $S$ could simply reflect that some lines are more motion-tuned. We run it through the same ridge readout and noise normalization, and the deep model must explain cell-type differences that this model does not.
The key ablation is the naive per-line analysis without noise-ceiling normalization and count matching. It shows how much apparent cell-type structure is just reliability, which POYO-CAP's findings predict will be substantial. All model variants receive the same training compute and the same readout hyperparameter search.
Why not just…?
Why not just split Bakhtiari's RSA by Cre line and report it? That is the obvious first step, and we run it as the ablation. It is not trustworthy on its own. POYO-CAP shows that Cre lines differ systematically in predictability, so lines will differ in raw alignment simply because some are noisier. Lines are also unevenly distributed across areas and depths, so a raw split confounds cell type with area. The mixed-effects, noise-normalized design exists to separate these factors.
Why not just use motion tuning? If PT-like lines are simply more direction-selective, a classical motion-energy model might reproduce any cell-type pattern without a dual-stream network. This is a real threat, and the motion-energy baseline tests it directly. If that baseline explains the cell-type component as well as ResNet-2p does, we will report that the cell-type effect is a low-level tuning difference, not evidence about stream computation.
Why not just use a neural foundation model like POYO+? POYO+ already separates cell types from activity, but it decodes from neurons to behavior. It cannot say which image-computable computation each type performs, which is the question here. Its embeddings could serve as a downstream check that lines we find dorsal-preferring also cluster together, but they cannot replace a pixel model.
Why not just model depth in a single network? Cell types differ by layer, and layer could map onto network depth rather than onto stream. ResNet-1p tests this directly. If its best-layer index explains the cell-type component equally well, the effect is hierarchical, not stream-specific.
Proposed Experiments and Decisive Checks
The first decisive check asks whether cell line explains stream preference beyond area in CPC ResNet-2p after reliability control. A second asks whether the sign is consistent within areas, with PT-like lines more dorsal than IT-like lines inside VISlm and inside VISam alike. A positive answer to both overturns the purely area-level reading. A null answer, with tight intervals, supports it at a resolution never previously tested.
The third check is the model-variant comparison. If the cell-type component appears only under contrastive predictive coding and vanishes under supervised training and in ResNet-1p, that bears directly on the objective-versus-architecture question raised by Choi et al. and Bakhtiari. Fourth, a stimulus-frame shuffle provides the null distribution for every effect.
Fifth, for Vip, Sst and Pvalb lines, we split trials by running speed to test whether locomotion shifts interneuron stream preference. Dipoppa's V1 results motivate this test, but they have not been extended to higher areas.
Success would mean a published, reproducible decomposition of stream alignment into area and cell type. It would come with a clear statement of which training regime, if any, produces cell-type-specific streams.
References
- The functional specialization of visual cortex emerges from training parallel pathways with self-supervised predictive learning
- A Dual-Stream Neural Network Explains the Functional Segregation of Dorsal and Ventral Visual Pathways in Human Brains
- Pyramidal cell types drive functionally distinct cortical activity patterns during decision-making
- From Individual to Population: Circuit Organization of Pyramidal Tract and Intratelencephalic Neurons in Mouse Sensorimotor Cortex
- Decoding Dynamic Visual Experience from Calcium Imaging via Cell-Pattern-Aware Pretraining
- Multi-session, multi-task neural decoding from distinct cell-types and brain regions
- Vision and Locomotion Shape the Interactions between Neuron Types in Mouse Visual Cortex