Abstract
Models that predict brain responses to stimuli provide one measure of understanding of a sensory system, and have many potential applications in science and engineering. Deep artificial neural networks have emerged as the leading such predictive models of the visual system, but are less explored in audition. Prior work provided examples of audio-trained neural networks that produced good predictions of auditory cortical fMRI responses and exhibited correspondence between model stages and brain regions, but left it unclear whether these results generalize to other neural network models, and thus how to further improve models in this domain. We evaluated model-brain correspondence for publicly available audio neural network models along with in-house models trained on four different tasks. Most tested models out-predicted previous filter-bank models of auditory cortex, and exhibited systematic model-brain correspondence: middle stages best predicted primary auditory cortex while deep stages best predicted non-primary cortex. However, some state-of-the-art models produced substantially worse brain predictions. Models trained to recognize speech in background noise produced better brain predictions than models trained to recognize speech in quiet, potentially because hearing in noise imposes constraints on biological auditory representations. The training task influenced the prediction quality for specific cortical tuning properties, with best overall predictions resulting from models trained on multiple tasks. The results generally support the promise of deep neural networks as models of audition, though they also indicate that current models do not explain auditory cortical responses in their entirety.
Introduction
In visual neuroscience, deep neural networks (DNNs) have become the standard for modeling hierarchical processing. However, the auditory system has remained less explored. While initial studies suggested that DNNs trained on speech and music could predict brain responses, it remained unclear if this alignment was a general property of audio architectures or a result of specific training conditions. This research addresses the "generality" problem by evaluating whether various audio DNNs—trained on different tasks and with different architectures—consistently mirror the human auditory cortex.
Methods and Data
The researchers conducted a large-scale comparative study using 19 different models. These included nine publicly available "external" models (Transformers, RNNs, and CNNs) and ten "in-house" models (CochCNN9 and CochResNet50) trained on combinations of four tasks: Word, Speaker, Environmental Sound (AudioSet), and Musical Genre recognition.
Two primary analytical methods were employed to evaluate model-brain correspondence:
- Voxelwise Regression (Encoding Analysis): Internal model activations were mapped to fMRI BOLD responses using ridge regression to determine if a linear combination of model "neurons" could predict human brain activity.
- Representational Similarity Analysis (RSA): This method compared the "geometry" of representations, checking if sounds that elicit similar activity in the brain also produce similar patterns in the model.
The brain data consisted of two independent fMRI datasets (NH2015 and B2021) from 28 participants listening to 165 natural sounds. Models were benchmarked against a traditional SpectroTemporal filter model.
Key Insights and Results
The study revealed several critical findings regarding the relationship between artificial and biological auditory processing:
- Hierarchical Alignment: A robust pattern emerged across successful models where middle layers best predicted the primary auditory cortex, while deeper layers aligned with non-primary areas. This suggests a shared "low-to-high" processing hierarchy between DNNs and the human brain.
- Task-Dependent Representation: The specific goal of a model dictates its internal structure. Models trained on speech tasks were significantly better at predicting speech-selective brain regions. Furthermore, models trained in noisy environments provided better brain predictions than those trained in quiet, suggesting that environmental constraints shape biological representations.
- The Multi-Task Advantage: The most "brain-like" model was a multi-task CochResNet50 trained simultaneously on words, speakers, and sounds. This suggests the human auditory cortex is optimized for general-purpose processing rather than isolated tasks.
- Engineering vs. Biology: High performance on engineering benchmarks (e.g., speech enhancement) did not always translate to biological realism. Some state-of-the-art models were poor predictors of brain activity, indicating that the brain's objective is not merely signal cleaning but specific information extraction.
Quantitatively, 15 of the 19 models outperformed the traditional baseline. The best-performing multi-task model explained 58% to 88% of the variance in specific cortical response components, such as those for pitch and speech.
Discussion and Future Directions
While the results support the use of DNNs as models for audition, current models do not fully capture auditory cortical responses. Limitations include the temporal resolution of fMRI compared to neural spiking and the current reliance on supervised learning. Future research may focus on branching architectures that mimic anatomical pathways and self-supervised learning objectives that better reflect how humans learn from their environment without explicit labels.
Summary updated