This paper presents an "in-silico audit" of TRIBE v2, a state-of-the-art foundation model designed to predict human brain activity (fMRI) from multimodal inputs like video and audio. While the model excels at identifying what is in a scene (e.g., faces or places), this study investigates whether it respects the fundamental "where" of the brain: its topographic maps.
TL;DR
The researchers found that while TRIBE v2 accurately localizes motion-selective responses in the brain, it completely fails to replicate the spatial (retinotopic) and frequency (tonotopic) maps found in primary sensory cortices. This suggests that current architectural choices, specifically spatial averaging in video encoders, create a "blind spot" for experiments depending on precise visual or auditory positioning.
What Problem Does This Paper Address?
In neuroscience, "topographic organization" refers to the way the brain maps the external world onto its surface. For example, adjacent points in your visual field are processed by adjacent neurons in the visual cortex (retinotopy). Similarly, different sound frequencies are mapped in a specific order in the auditory cortex (tonotopy).
Foundation models like TRIBE v2 are increasingly used as "digital twins" for the brain to run virtual experiments. However, if these models don't capture these basic organizational principles, they cannot be trusted for any in-silico research that depends on spatial or frequency-specific information. The author sought to quantify exactly where these models succeed and where they break down.
Methods
The author treated the TRIBE v2 model like a human subject in an fMRI scanner, using a specialized "audit" pipeline:
- Stimuli: The model was "shown" classic mapping stimuli, including rotating checkerboard wedges, expanding rings, and logarithmic tone sweeps.
- Analysis: Standard fMRI analysis techniques, such as population receptive field (pRF) fitting and Fourier analysis, were applied to the model's predicted vertex-wise responses.
- Controls: To ensure the analysis pipeline was working, the author created a "stand-in" model with known ground truth (both a positive control that had maps and a negative control that didn't).
- Statistical Validation: The resulting brain maps were compared against established human brain atlases using spin tests, which rotate maps on a sphere to see if their spatial layout matches real anatomy by chance.
Datasets
The study did not involve new human data collection. Instead, it used:
- Model: The released
facebook/tribev2checkpoint (Video encoder: V-JEPA 2; Audio encoder: Wav2Vec-BERT 2.0). - Reference Atlases: The Benson retinotopy template (for V1-V3) and the Glasser HCP-MMP parcellation (for area MT+).
- Synthetic Stimuli: Custom-generated video and audio files designed to trigger specific topographic responses.
Key Insights and Ideas
- The "What" vs. "Where" Split: The model shows a sharp divide. It can identify features like "motion" regardless of where they appear, but it loses the coordinate system of the visual field.
- Architectural Determinism: The failure of retinotopy was predicted by the model's architecture. TRIBE v2 averages video features over space before they reach the "response head." This effectively "blurs" the positional information required to build a map.
- The Tonotopy Mystery: Surprisingly, the model also failed to show a frequency map (tonotopy) in the auditory cortex, even though its audio encoder does not use the same spatial pooling. This suggests the limitation might stem from the training data (mostly speech and film) rather than just the architecture.
- Global Stats vs. True Maps: The model showed a weak "eccentricity-like" signal, but further testing revealed this was likely an artifact of global image statistics (like total brightness or contrast) rather than a genuine spatial code.
Key Results
- Motion Selectivity (Success): The model correctly localized motion-selective responses to area MT+. In fact, 22% of the most motion-sensitive vertices fell within this tiny 0.42% sliver of the cortex—a 50-fold enrichment.
- Retinotopy (Failure): There was virtually no contralateral response (the left brain didn't prefer the right visual field) and no coherent polar-angle map. The correlation with human templates was near zero ($r \approx -0.02$).
- Tonotopy (Failure): The "best frequency" spread across the auditory cortex was only 0.14 octaves, compared to the 2.0 octaves expected in a real brain.
Limitations and Future Work
- Resolution: The study used the
fsaverage5mesh, where vertices are ~4mm apart. This might be too coarse to catch very fine-grained structures. - Average Subject: TRIBE v2 predicts a group-average response, which naturally smooths out the idiosyncratic maps found in individuals.
- Future Directions: The author suggests that future versions of brain-response models must preserve spatial structure through to the final prediction layer if they are to be used for position-dependent in-silico neuroscience. Testing different audio encoders or training on more diverse sounds might also resolve the tonotopy issue.
Summary updated