Representational Alignment between Deep Neural Networks and Human Brain in Speech Processing under Audiovisual Noise

Cheng Luo, Jian Zhu, Zhoujian Sun, Yunyi Qi, Wenhui Sun, Wenyuan YuarXiv 2026 · arXiv

This study explores the fascinating intersection of artificial intelligence and neuroscience, asking a fundamental question: Do the internal representations of state-of-the-art speech recognition models mirror the hierarchical processing of the human brain, especially when faced with the "cocktail party" problem of noisy, audiovisual environments?

TL;DR

Researchers found that transformer-based speech models (AV-HuBERT) exhibit a hierarchical alignment with the human brain, where shallow layers process acoustic features and deep layers handle linguistic information. Crucially, fine-tuning these models on noisy audiovisual data enhances this "brain-likeness," improving noise disentanglement in early layers and refining abstract linguistic representations in later ones.

What Problem Does This Paper Address?

While Deep Neural Networks (DNNs) have achieved human-level performance in Automatic Speech Recognition (ASR), it remains unclear if they achieve this through human-like internal computations. Previous research suggested a general correlation between DNN hierarchies and the auditory pathway, but several gaps remained:

  1. Level of Analysis: Most studies didn't distinguish between basic acoustic processing (like sound envelopes) and higher-level linguistic processing (like word surprisal).
  2. Environmental Realism: Real-world speech is rarely "clean." It involves background noise and visual cues (lip movements, scene context).
  3. The Role of Learning: It was unknown how the process of fine-tuning a model for a specific task (like recognizing speech in noise) changes its alignment with the brain.

Methods

The authors developed a novel framework combining Neural Encoding Models and Representational Similarity Analysis (RSA) to compare a transformer-based model (AV-HuBERT) with human brain activity recorded via EEG.

Figure 1: Schematic of analysis framework for computing representational similarity between DNNs embeddings and neural responses. For transformer-based DNNs (top panel), the model is fine-tuned on speech data with congruent audiovisual noise, starting from a pre-trained initialization. Then, embedding vectors of stimulus input are extracted from 12 transformer hidden layers, and utilized to calculate representational dissimilarity matrixs (RDMs). For neural encoding model (bottom panel), the EEG data in each EEG channel is predicted based on a linear combination of time-shifted stimulus features (speech envelope, noise envelope, word onset, and word surprisal). The encoder weights (temporal response functions) are estimated by maximizing the correlation between the actual neural responses and simulated neural responses. The simulated neural responses to stimulus features are utilized to calculate RDMs. Representational similarity is index through Spearman correlation between the RDMs of model embeddings and simulated neural responses.
  • The Brain Side: EEG was recorded from 20 participants listening to speech clips overlaid with real-world noise (e.g., street sounds, playgrounds). The researchers used Multivariate Temporal Response Functions (mTRFs) to isolate neural responses to specific features:
  • Acoustic: Speech envelope and noise envelope.
  • Linguistic: Word onset and word surprisal (how "unexpected" a word is).
  • The Model Side: They used the AV-HuBERT architecture, extracting embedding vectors from its 12 transformer layers. They compared two versions: a "pre-trained" model and one "fine-tuned" on noisy audiovisual data.
  • The Comparison: RSA was used to quantify the similarity between the model's internal "view" of the stimuli and the brain's "view" at different processing levels.

Datasets

  • Speech: LRS3-TED corpus (English speech).
  • Noise: AVNS corpus (real-world audiovisual noise like roads and playgrounds).
  • Pre-training: VoxCeleb2 (1,326 hours) and LRS3 (433 hours).
  • Fine-tuning: A synthetic 433-hour dataset where LRS3 speech was mixed with AVNS noise at a 0 dB signal-to-noise ratio.

Key Insights and Ideas

  • Hierarchical Mirroring: The study confirms a clear "shallow-to-deep" transition. Shallow DNN layers correlate most strongly with the brain's acoustic processing, while deeper layers align with linguistic processing.
  • Fine-tuning as a "Humanizer": Fine-tuning isn't just about boosting accuracy (which jumped from 0% to 77% in noise); it actually forces the model to adopt a more brain-like strategy.
  • Noise Disentanglement: One of the most striking "aha" moments was that fine-tuning shifted the trend for noise processing. It increased noise representation in shallow layers, suggesting the model learns to "filter" or disentangle noise early on, allowing deeper layers to focus on cleaner linguistic extraction—much like the human auditory system.

Key Results

  • Functional Alignment: In audiovisual-congruent conditions (where video matches the noise), significant brain-model similarity was found across almost all layers. However, this similarity vanished in "incongruent" conditions for the fine-tuned model, suggesting the model (like the brain) becomes highly specialized for congruent environmental context.
  • Feature-Specific Mapping:
    • Acoustic features (speech/noise envelopes) peaked in similarity within layers 1–5.
  • Linguistic features (word onset/surprisal) showed increasing similarity toward the deeper layers, peaking around layer 11.
  • Shift in Dominance: Fine-tuning caused the transition from acoustic-dominance to linguistic-dominance to happen earlier in the network (around layer 6) compared to the pre-trained model (around layer 8).
Figure 3: Topographic maps of representational similarity between DNN embeddings and simulated neural responses to speech envelope, noise envelope, word onset, and word surprisal in speech processing with audiovisual-congruent noise. Results for pre-trained model (A), and fine-tuned model (B). Black dots denote the significant channels.

Limitations and Future Work

  • Linguistic Diversity: The study only involved native English speakers and English stimuli. It remains to be seen if these hierarchical alignments hold across typologically different languages.
  • Architecture Specificity: The results are based on AV-HuBERT. While the authors expect similar patterns in other transformers like Wav2Vec 2.0, this needs empirical verification.
  • Visual Integration: While the model is audiovisual, the neural encoding focused on acoustic and linguistic features. Future work could explicitly model the neural encoding of visual features (like lip movements) to see how they integrate into the DNN hierarchy.

Summary updated

Abstract

Speech comprehension in the human brain is an incremental process that begins with acoustic processing and advances to linguistic processing. While recent studies have revealed that the hierarchy of deep neural networks (DNNs) correlates with the ascending auditory pathway, the exact nature of this DNN-brain alignment remains underexplored. In this study, we investigate how DNN representations align with the brain's acoustic-to-linguistic processing. Specifically, we employed neural encoding models to simulate neural responses to acoustic (i.e., speech and noise envelope) and linguistic features (i.e., word onset and surprisal). By applying representational similarity analysis (RSA), we quantified the similarity between these neural responses and the DNN embeddings generated by a pre-trained automatic speech recognition (ASR) model, both before and after fine-tuning on audiovisual noisy data. Our results demonstrate significant DNN-brain alignment: embeddings from shallow layers exhibit higher similarity to neural responses associated with acoustic features, while those from deeper layers align more closely with neural responses related to linguistic features. Importantly, the fine-tuning process enhances this alignment by improving noise processing in shallower layers and refining linguistic representations in deeper layers. These results suggest that fine-tuned DNN models can naturally develop human-like processing patterns in noisy environments, highlighting a functional alignment between the human brain and DNNs in speech representation.
BibTeX
@misc{luo2026representational,
  author = {Cheng Luo and Jian Zhu and Zhoujian Sun and Yunyi Qi and Yu Qi and Wenhui Sun and Wenyuan Yu},
  title = {{Representational Alignment between Deep Neural Networks and Human Brain in Speech Processing under Audiovisual Noise}},
  year = {2026},
  eprint = {2611.12291},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.12291},
}
Figure