Introduction
Artificial Neural Networks (ANNs) were conceptually inspired by biological neural systems, yet the development of these two fields has largely diverged. While Convolutional Neural Networks (CNNs) excel at visual recognition, the internal logic governing their filters often remains opaque. Conversely, functional Magnetic Resonance Imaging (fMRI) provides a window into human brain activity but often lacks the semantic context needed to map specific neural responses to complex visual features.
To bridge this gap, researchers have developed Synchronized Activations (Sync-ACT), a framework that aligns the internal representations of AI models with human brain activity recorded during naturalistic movie-watching. By using time as a common axis, Sync-ACT enables the "cross-annotation" of artificial neurons with biological descriptions and reveals a profound link between an AI's biological plausibility and its task performance.
Methodology: The Sync-ACT Framework
The core of Sync-ACT is the synchronization of two distinct data streams—artificial and biological—using a shared naturalistic stimulus.
1. Temporal Activation Extraction
Both the AI model and human subjects are presented with the same movie sequences. For the AI, movie frames are processed by pre-trained CNNs (e.g., ResNet, DenseNet). The maximum activation of each convolutional filter is recorded at each time step, generating a "temporal activation series" for every artificial neuron.
2. Functional Brain Network (FBN) Encoding
On the biological side, 4D fMRI data is processed using a deep learning Encoder-Decoder architecture equipped with Multi-Head Self-Attention. This model compresses complex brain signals into 64 distinct Functional Brain Networks (FBNs), each with its own temporal activation series.
3. Coupling and Cross-Annotation
The framework calculates the Pearson Correlation Coefficient (PCC) between the AI's filter activations and the brain's FBN activations. High PCC values indicate synchronization. Once paired, AI filters are annotated with human-readable semantic labels (e.g., "face recognition," "navigation") derived from established neuroscientific meta-analyses of the corresponding FBNs.
Key Results and Discussion
The application of Sync-ACT across multiple datasets, including the HCP 7T and StudyForrest projects, yielded several significant insights into the relationship between artificial and biological vision.
The Brain-Performance Link
A pivotal discovery of this research is the strong linear relationship between an AI model's similarity to the human brain and its objective performance. As shown in the figure below, there is a high correlation ($R^2 \approx 0.75$) between the average PCC (brain-sync score) and Top-1 accuracy on ImageNet classification.
This suggests that architectures which more closely approximate biological visual processing strategies tend to be more effective at real-world recognition tasks. Advanced models like DenseNet-161 and ResNet-50 consistently showed higher brain-similarity scores compared to older architectures like VGG-16.
Hierarchical Alignment
The study also confirmed a hierarchical alignment: higher-level CNN layers, which process complex semantic concepts, show significantly stronger synchronization with the human brain than lower-level layers responsible for basic edges and textures. This mirrors the known hierarchical organization of the human visual cortex.
Conclusion and Future Directions
Sync-ACT provides a novel methodology for grounding AI interpretability in biological reality. By demonstrating that "brain-like" models perform better, this work opens the door for using brain-similarity as a metric in Neural Architecture Search (NAS) to design the next generation of bio-inspired AI. Future work will look to expand this paradigm to Vision Transformers (ViTs) and incorporate finer-grained natural language descriptions for even more precise semantic mapping.
Summary updated