Introduction
In the field of "brain scoring," researchers aim to align the internal representations of deep learning models with biological brain activity. Traditionally, this has been approached at a "coarse-grained" level, where entire transformer layers are compared to neural responses. However, a single layer in a model like Wav2Vec 2.0 is a complex assembly of multiple attention heads, each processing different contextual information. Treating the whole layer as a single unit masks the nuanced ways the brain disentangles acoustic signals. This paper introduces a "fine-grained" approach, treating individual hidden dimensions as distinct artificial neurons (ANs), revealing a more precise mapping to the human auditory cortex and identifying a previously overlooked "transient stage" in speech processing.
Methodology
The researchers developed a framework to define and validate neuron-level Artificial Neurons (ANs) and their biological counterparts.
Defining Artificial Neurons (ANs)
Instead of using aggregate layer-level embeddings, the authors looked inside the Multi-Head Self-Attention (MSA) modules. An AN is defined as a single hidden dimension within the element-wise product of the "queries" (Q) and "keys" (K). This shifts the granularity from 12 layers to over 9,000 individual ANs, allowing for a much more detailed analysis of the model's internal logic.
Defining Biological Neurons (BNs)
On the biological side, to avoid noisy voxel-level data, the study employed Volumetric Sparse Deep Belief Networks (VS-DBN) to identify Functional Brain Networks (FBNs). Each FBN is treated as a single "Biological Neuron" (BN).
The Synchronization Bridge
Using the "Narratives" fMRI dataset (specifically the "Pieman" and "The Man Who Forgot Ray Bradbury" sessions), the researchers played spoken stories to both human subjects and the Wav2Vec 2.0 model. Pearson correlation was used to identify synchrony between AN "firing" and BN activity. To validate these mappings, the authors also performed phoneme mapping to ensure specific ANs were responding to meaningful speech sounds.
Results and Discussion
The fine-grained analysis yielded several key insights into the alignment between computational and biological hierarchies.
The "Transient Stage" and Hierarchy Alignment
A significant discovery is the "transient stage" in processing. Layers 1-5 of Wav2Vec 2.0 are strongly anchored to a "Global BN" (BN#47), centered in Heschl’s gyrus (HG) and the Superior Temporal Gyrus (STG), which handles primary acoustic and phonetic processing. As processing moves to layers 11-12, representations become highly localized and specific.
Layers 6-10 act as a bridge, gradually integrating phonetic information with higher-level language networks, such as Broca’s area. This computational hierarchy aligns with the cortical hierarchy of speech processing: moving from primary auditory areas to semantic and working memory regions.
Synchronization and Functional Specificity
The study found that 99.86% of the proposed ANs showed statistically significant correlations with brain activity ($p < 0.01$). Maximum Pearson correlation coefficients reached between 0.33 and 0.42, significantly higher than inter-session baselines. Furthermore, specific ANs were found to be "tuned" to certain phoneme categories (e.g., vowels vs. fricatives), synchronizing with brain regions known for those specific linguistic tasks.
Conclusion
By zooming in to the neuron level, this research provides a much clearer picture of how audio-transformers mimic human auditory encoding. The identification of the "transient stage" and the "Global BN" highlights the sophisticated disentangling process occurring in both artificial and biological systems. While limited by the temporal resolution of fMRI, this work sets the stage for future studies using high-speed temporal dynamics to further explore the alignment between AI and the human brain.
Summary updated