End-to-end neural system identification with neural information flow

Katja Seeliger, Luca Ambrogioni, Yagmur Güçlütürk, L. M. van den Bulk, Umut Güçlü, Marcel A. J. van GervenPLOS Computational Biology 2021 · DOI

Introduction

Understanding how the human brain processes complex sensory input is a fundamental goal of neuroscience. Traditionally, researchers have employed "system identification" to build mathematical models that replicate neural responses. While Deep Neural Networks (DNNs) have emerged as the leading candidates for these models, most are "task-driven"—pre-trained on artificial datasets like ImageNet. This introduces biases that may not align with biological processing.

This paper introduces Neural Information Flow (NIF), a data-driven framework designed to build "synthetic brains" by training models directly on real brain activity. By leveraging a massive dataset of fMRI recordings from a participant watching 23 hours of the series Doctor Who, the researchers demonstrate that biological features can emerge autonomously without manual labeling or task-specific pre-training.

The Neural Information Flow (NIF) Framework

The NIF framework moves away from black-box AI by implementing a modular, biologically constrained architecture. Unlike standard DNNs, NIF modules correspond one-to-one with specific anatomical regions of the visual cortex, such as V1, V2, V3, the Fusiform Face Area (FFA), and the Middle Temporal (MT) area.

The NIF architecture consists of hierarchical modules mapped to specific brain regions, utilizing spatiotemporal convolutions and factorized observation models to predict fMRI signals.

The model utilizes three core components:

  • Modular Architecture: Interconnected tensors represent distinct brain regions.
  • Spatiotemporal Convolutions: These model the spatial layout of neurons and the temporal delays inherent in synaptic transmission.
  • Factorized Observation Models: To bridge the gap between internal model states and fMRI voxels, NIF decomposes responses into spatial (where), temporal (when), and feature (what) components.

The entire system is trained end-to-end using stochastic gradient descent to minimize the discrepancy between predicted and observed BOLD signals.

Experimental Design and Datasets

The study utilized an exceptionally rich dataset consisting of high-resolution 3T fMRI recordings. A single participant viewed 30 episodes (~23 hours) of Doctor Who. This naturalistic stimulus provides a diverse range of visual information, from low-level edges to high-level social interactions. To validate the model's generalizability, it was tested on separate short films (Pond Life and Space/Time) that were not included in the training set.

Key Results and Emergent Biology

The NIF model successfully predicted BOLD responses across the majority of voxels in the targeted visual areas. Notably, the model's predictive power peaked after approximately 12 hours of training data, highlighting the necessity of long-duration recording sessions for deep neural modeling of the brain.

Prediction performance of the NIF model. (A) Histograms show the distribution of correlation coefficients (r) across different regions of interest. (B) The cortical map illustrates the spatial distribution of prediction accuracy (z(r)).

Several biological phenomena emerged spontaneously during training:

  • Feature Detection: The V1 module developed Gabor-like edge detectors, while the FFA module learned to respond specifically to faces, despite no explicit instruction to do so.
  • Retinotopic Mapping: The model recovered accurate maps of the visual field (eccentricity and polar angle) directly from natural video stimuli, a task that usually requires specialized flickering stimuli.
  • Superior Performance: In several regions, the data-driven NIF model outperformed state-of-the-art task-driven models trained on massive video datasets like Kinetics.

Discussion and Future Directions

While NIF represents a significant step toward biologically plausible neural modeling, challenges remain. The current framework is primarily feed-forward, whereas the biological brain relies heavily on feedback loops and lateral connections. Additionally, the "data hunger" of such models poses a logistical challenge for standard fMRI protocols. Future iterations of NIF may incorporate recurrent neural architectures to better capture the dynamic nature of human cognition and the complex interplay between brain regions.

Summary updated

BibTeX
@article{seeliger2021end,
  author = {Katja Seeliger and Luca Ambrogioni and Yagmur Güçlütürk and L. M. van den Bulk and Umut Güçlü and Marcel A. J. van Gerven},
  title = {{End-to-end neural system identification with neural information flow}},
  journal = {PLOS Computational Biology},
  year = {2021},
  doi = {10.1371/journal.pcbi.1008558},
}
Figure