The neural processing of hierarchical structure in music and speech at different timescales

Morwaread Farbood, David J. Heeger, Gary Marcus, Uri Hasson, Yulia LernerFrontiers in Neuroscience 2015 · DOI

Abstract

Music, like speech, is a complex auditory signal that contains structures at multiple timescales, and as such is a potentially powerful entry point into the question of how the brain integrates complex streams of information. Using an experimental design modeled after previous studies that used scrambled versions of a spoken story and a silent movie, we investigate whether listeners perceive hierarchical structure in music beyond short (~6 s) time windows and whether there is cortical overlap between music and language processing at multiple timescales. Experienced pianists were presented with an extended musical excerpt scrambled at multiple timescales—by measure, phrase, and section—while measuring brain activity with functional magnetic resonance imaging (fMRI). The reliability of evoked activity, as quantified by inter-subject correlation of the fMRI responses, was measured. We found that response reliability depended systematically on musical structure coherence, revealing a topographically organized hierarchy of processing timescales. Early auditory areas responded reliably in all conditions. For brain areas at the top of the hierarchy, the original (unscrambled) excerpt evoked more reliable responses than any of the scrambled excerpts, indicating that these brain areas process long-timescale musical structures, on the order of minutes. The topography of processing timescales was analogous with that reported previously for speech, but the timescale gradients for music and speech overlapped with one another only partially, suggesting that temporally analogous structures—words/measures, sentences/musical phrases, paragraph/sections—are processed separately.

Introduction

A central debate in neuroscience is whether the brain integrates long-term musical structures—which unfold over minutes—similarly to how it handles discourse in language. While we understand how the brain processes "local" features like pitch or rhythm, it remains unclear if music is primarily consumed in short windows or if it engages higher-order circuits for long-term integration. This study seeks to map the "Temporal Receptive Windows" (TRWs) of the brain—the specific durations of information a neural circuit can integrate—to determine how the brain scales its processing of musical logic.

Methods

Researchers recruited 18 experienced pianists to undergo fMRI while listening to a 4-minute recording of Brahms’ Piano Concerto No.

  1. To test the brain's reaction to varying levels of musical coherence, the stimulus was scrambled at four hierarchical levels:
  2. Intact: The original recording.
  3. Sections: Scrambled at ~38-second intervals.
  4. Phrases: Scrambled at ~6-second intervals.
  5. Measures: Scrambled at ~1-second intervals.
  6. Backward: Audio played in reverse to remove all musical structure.
The musical stimulus was scrambled at different hierarchical levels: measures (yellow), phrases (green), and sections (blue).

The team utilized Inter-Subject Correlation (ISC) to analyze the data, measuring the degree to which brain activity synchronized across participants. High synchronization in a specific condition indicates that the brain region reliably processes information at that timescale.

Results and Key Insights

The study revealed a clear topographic hierarchy of processing timescales. Early auditory areas (A1+) exhibited short TRWs, syncing even to backward music. However, as processing moved toward the frontal and parietal lobes, the TRWs lengthened, requiring coherent phrases or the full intact piece to show reliable activity.

Brain maps illustrating the topographic hierarchy of processing timescales, from early auditory areas to higher-order regions like the precuneus and mPFC.

Key findings include:

  • Topographic Gradient: The Precuneus and medial Prefrontal Cortex (mPFC) sat at the top of the hierarchy, responding reliably only to the intact musical excerpt.
  • Speech vs. Music Divergence: While both music and speech follow a hierarchical map, they utilize different neural real estate. The "music axis" and "speech axis" only partially overlapped in the Superior Temporal Gyrus, with higher-order areas processing musical structure and linguistic meaning in distinct regions.
  • Emotional Syncing: Subcortical areas like the Nucleus Accumbens and Amygdala only showed reliable syncing for long-term coherent versions, suggesting that the emotional payoff of music requires a structural arc.

Discussion

These results suggest that the brain treats the structural logic of music and language as distinct phenomena, despite sharing a similar hierarchical organization. While experienced pianists show robust long-term integration, future work is needed to determine if non-musicians exhibit the same hierarchical syncing. Additionally, further research could explore if different musical genres, such as ambient or repetitive music, engage these long-timescale circuits differently.

Summary updated

Abstract

Music, like speech, is a complex auditory signal that contains structures at multiple timescales, and as such is a potentially powerful entry point into the question of how the brain integrates complex streams of information. Using an experimental design modeled after previous studies that used scrambled versions of a spoken story (Lerner et al., 2011) and a silent movie (Hasson et al., 2008), we investigate whether listeners perceive hierarchical structure in music beyond short (~6 s) time windows and whether there is cortical overlap between music and language processing at multiple timescales. Experienced pianists were presented with an extended musical excerpt scrambled at multiple timescales-by measure, phrase, and section-while measuring brain activity with functional magnetic resonance imaging (fMRI). The reliability of evoked activity, as quantified by inter-subject correlation of the fMRI responses, was measured. We found that response reliability depended systematically on musical structure coherence, revealing a topographically organized hierarchy of processing timescales. Early auditory areas (at the bottom of the hierarchy) responded reliably in all conditions. For brain areas at the top of the hierarchy, the original (unscrambled) excerpt evoked more reliable responses than any of the scrambled excerpts, indicating that these brain areas process long-timescale musical structures, on the order of minutes. The topography of processing timescales was analogous with that reported previously for speech, but the timescale gradients for music and speech overlapped with one another only partially, suggesting that temporally analogous structures-words/measures, sentences/musical phrases, paragraph/sections-are processed separately.
BibTeX
@inproceedings{farbood2015neural,
  author = {Morwaread Farbood and David J. Heeger and Gary Marcus and Uri Hasson and Yulia Lerner},
  title = {{The neural processing of hierarchical structure in music and speech at different timescales}},
  booktitle = {Frontiers in Neuroscience},
  year = {2015},
  doi = {10.3389/fnins.2015.00157},
}
Figure