Distinct social-linguistic processing between humans and large audio-language models: Evidence from model-brain alignment

Hanlin Wu, Xufeng Duan, Zhenguang CaiNAACL 2025

TL;DR

This paper investigates how large audio-language models (LALMs) process social and biological incongruities in speech, comparing their "understanding" to human brain responses (EEG). While one LALM (Qwen2-Audio) shows some alignment with human N400 responses to social violations, neither model replicates the human distinction between social (N400) and biological (P600) violations, suggesting current LALMs lack human-like rational inference mechanisms for real-time language processing.

What Problem Does This Paper Address?

Humans are remarkably adept at using speaker characteristics (like gender, age, personality) to contextualize and comprehend spoken language in real-time. When a speaker's attributes clash with the content of their speech – for instance, a man saying "The first time I got pregnant" (biological impossibility) or "I like to get manicures" (social stereotype violation) – human brains show distinct electrical responses. Specifically, social stereotype violations elicit an N400 brainwave, while biological impossibilities trigger a P600 wave. This suggests humans engage in a form of "rational inference," using world knowledge to interpret or correct perceived inconsistencies.

With the rise of sophisticated Large Audio-Language Models (LALMs) that can process audio, a crucial question emerges: do these models process speaker-contextualized language in a way that mirrors human cognitive mechanisms? Understanding this is vital for building more natural and trustworthy human-AI interactions, as well as for gaining insights into the emergent cognitive abilities of these models. This paper aims to bridge this gap by comparing LALM processing patterns with human EEG data.

Methods

The researchers employed a two-pronged approach:

  1. Human EEG Data: They utilized existing EEG data from a previous study (Wu and Cai, 2024b) where Mandarin Chinese speakers listened to sentences that were either speaker-congruent or speaker-incongruent, categorized as either social or biological violations. The EEG data captured brain responses (N400 and P600 effects) to these stimuli.
  2. LALM Metrics: They selected two LALMs: Qwen2-Audio 7B Instruct and Ultravox 0.5 8B. For these models, they extracted computational metrics – specifically, surprisal and entropy – at the critical word of each sentence.
  3. Surprisal measures the unpredictability of a word given its context. Higher surprisal indicates a word is less expected, often linked to increased processing effort.
  4. Entropy quantifies the uncertainty in the model's probability distribution for the next word. Higher entropy suggests more uncertainty in the model's prediction.

    These metrics were collected for both original Chinese and translated English versions of the stimuli, using both audio and text-based inputs to the models.

The core of the analysis involved comparing these LALM metrics with the human EEG responses. They used linear mixed-effects (LME) modeling to assess:

  • Whether LALM surprisal and entropy aligned with human processing patterns (i.e., whether they showed similar responses to congruent vs. incongruent, social vs. biological conditions).
  • Whether LALM metrics could predict human brain responses (N400 and P600 amplitudes).

Datasets

The study used a dataset of 80 self-referential sentences in Mandarin Chinese, each with speaker-congruent and speaker-incongruent audio versions, categorized into social and biological violation types. An English translation of these sentences was also created. This dataset was originally collected for human EEG experiments and was repurposed for LALM analysis. No new benchmarks were introduced; instead, existing human EEG data served as the benchmark.

Key Insights and Ideas

  • Qwen2-Audio shows some promise: This model demonstrated increased surprisal for speaker-incongruent content and its surprisal values significantly predicted human N400 responses, suggesting some alignment with human processing of social violations.
  • Ultravox 0.5 lags behind: This model showed limited sensitivity to speaker characteristics and did not reliably predict human neural responses, indicating a weaker grasp of speaker-contextualized language.
  • The N400/P600 distinction is key: Humans show distinct brain responses (N400 for social, P600 for biological violations), reflecting a rational inference process where they distinguish between social norms and physical impossibilities.
  • LALMs miss the human nuance: Crucially, neither Qwen2-Audio nor Ultravox 0.5 replicated this human-like distinction. They treated social and biological violations similarly, suggesting they don't engage in the same kind of rational inference or error correction.
  • Surprisal predicts N400, not P600: Qwen2-Audio's surprisal was a good predictor of human N400 responses (associated with social violations) but not P600 responses (associated with biological violations). This further highlights the model's inability to capture the distinct processing of biological impossibilities.
  • Entropy is less informative: The models' predictive uncertainty (entropy) was primarily driven by linguistic properties of the input, not speaker-content relationships, and generally did not predict human neural responses.

Key Results

  • Qwen2-Audio: Showed a significant main effect of Congruency on surprisal (p = .037), indicating sensitivity to speaker-content incongruency. Its surprisal significantly predicted human N400 responses (p < .001). However, it did not show a significant interaction between Congruency and Type, meaning it didn't differentiate between social and biological violations.
  • Ultravox 0.5: Did not show a significant main effect of Congruency on surprisal, suggesting it was not sensitive to speaker-content incongruency. Its surprisal did not reliably predict human N400 or P600 responses.
  • Entropy: For both models, entropy did not show significant effects related to Congruency or Type, nor did it reliably predict human N400 or P600 responses.

Limitations and Future Work

The authors acknowledge several limitations:

  • Limited Model Scope: The study only examined two LALMs, which may not represent the full spectrum of current audio-language models or their diverse architectures and training paradigms.
  • Metric Sufficiency: Surprisal and entropy, while established metrics, might not fully capture the complex processing distinctions humans exhibit.
  • Static Speaker Processing: The study examined models' responses to single utterances, not their ability to dynamically adapt to an individual speaker's established patterns over time, a key aspect of human processing.

Future research directions suggested include:

  • Broader Model Evaluation: Investigating a wider range of LALMs and exploring alternative metrics, such as analyzing activation patterns in different model layers or using representation similarity analysis.
  • Dynamic Adaptation: Developing methods to assess how LALMs adapt their predictions based on multiple utterances from the same speaker, mirroring human listeners' ability to learn and adjust to individual speaking styles and biases.
  • Ethical Implications: Further exploring potential biases in LALMs, such as gender stereotypes, and their implications for real-world applications.

Summary updated

BibTeX
@inproceedings{wu2025distinct,
  author = {Hanlin Wu and Xufeng Duan and Zhenguang Cai},
  title = {{Distinct social-linguistic processing between humans and large audio-language models: Evidence from model-brain alignment}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
}
Figure