Modulating Cortical Instrument Representations During Auditory Stream Segregation and Integration With Polyphonic Music

Lars Gutschalk Hausfeld, Niels R. Disbergen, Giancarlo Valente, Robert J. Zatorre, Elia FormisanoFrontiers in Neuroscience 2021 · DOI

Abstract

Numerous neuroimaging studies demonstrated that the auditory cortex tracks ongoing speech and that, in multi-speaker environments, tracking of the attended speaker is enhanced compared to the other irrelevant speakers. In contrast to speech, multi-instrument music can be appreciated by attending not only on its individual entities (i.e., segregation) but also on multiple instruments simultaneously (i.e., integration). We investigated the neural correlates of these two modes of music listening using electroencephalography (EEG) and sound envelope tracking. To this end, we presented uniquely composed music pieces played by two instruments, a bassoon and a cello, in combination with a previously validated music auditory scene analysis behavioral paradigm (Disbergen et al., 2018). Similar to results obtained through selective listening tasks for speech, relevant instruments could be reconstructed better than irrelevant ones during the segregation task. A delay-specific analysis showed higher reconstruction for the relevant instrument during a middle-latency window for both the bassoon and cello and during a late window for the bassoon. During the integration task, we did not observe significant attentional modulation when reconstructing the overall music envelope. Subsequent analyses indicated that this null result might be due to the heterogeneous strategies listeners employ during the integration task. Overall, our results suggest that subsequent to a common processing stage, top-down modulations consistently enhance the relevant instrument's representation during an instrument segregation task, whereas such an enhancement is not observed during an instrument integration task. These findings extend previous results from speech tracking to the tracking of multi-instrument music and, furthermore, inform current theories on polyphonic music perception.

Introduction: The Challenge of Polyphonic Music

In complex auditory environments, such as a crowded room, the human brain possesses a remarkable ability to isolate a single voice—a phenomenon known as the "cocktail party effect." While this mechanism has been extensively studied in the context of speech, music presents a distinct cognitive challenge. Unlike speech, where background noise is typically ignored, music often requires listeners to oscillate between focusing on a single melodic line (segregation) and appreciating the collective harmony of an ensemble (integration).

This study investigates how the brain manages these two listening modes. By employing electroencephalography (EEG) and "sound envelope tracking," researchers sought to determine if the brain tracks a musical "aggregate" similarly to a single instrument and how top-down attention modulates these cortical representations over time.

Experimental Design and Methodology

The researchers recruited 19 adult non-musicians (15 of whom provided usable EEG data) to participate in a behavioral task involving custom-composed polyphonic pieces.

Experimental design showing musical stimuli with triplet variations and the trial/block structure.

Stimuli and Task

Participants listened to 20 unique, two-part counterpoint pieces (28 seconds each) featuring a bassoon and a cello. The instruments were equalized for Root Mean Square (RMS) power to ensure acoustic balance. The primary task involved a "rhythmic triplet" detection paradigm. As shown in Figure 1, these triplets could appear in the upper voice (bassoon), lower voice (cello), or cross between them. Participants were instructed to attend to either the bassoon, the cello, or the "aggregate" (both instruments) and report the presence of these triplets.

Neural Tracking Analysis

The study utilized sound envelope tracking to analyze 63-channel EEG data. Using a deconvolution approach (mTRF toolbox), the team reconstructed the amplitude envelopes of the instruments from the neural signals. They employed both multi-delay models to capture overall signal tracking and single-delay models to pinpoint specific latencies where attentional effects were most pronounced.

Results: Neural Tracking and Attentional Modulation

The analysis revealed several key insights into how attention shapes musical perception:

Segregation Success

During the segregation task, attended instruments were reconstructed significantly better than unattended ones ($r_z = 0.107$ vs. $r_z = 0.096$, $p = 0.020$). This "boost" in neural representation was particularly robust for the bassoon. While the cello showed a similar trend, its multi-delay reconstruction difference was less statistically significant on its own, though single-delay analysis revealed a clear window of attentional modulation between 150–210 ms.

Temporal Dynamics of Attention

Top-down modulation was not instantaneous. Significant attentional effects emerged in middle-latency (150–210 ms) and late-latency (320–450 ms) windows. This suggests that initial sound processing is primarily acoustic and bottom-up, with attention-driven feedback loops engaging later in the processing stream.

The Integration Null Result

Surprisingly, when participants were asked to listen to the "aggregate" music (integration), no significant overall attentional enhancement was observed in the reconstructed envelopes. Further investigation suggested this was due to heterogeneous strategies among listeners; some participants continued to track the bassoon primarily, while others balanced their attention between both instruments.

Upper-Voice Dominance

Regardless of the focus of attention, the bassoon (the higher-pitched instrument) was generally reconstructed more accurately than the cello. This finding aligns with the "upper voice dominance" effect frequently noted in music perception literature.

Discussion and Conclusion

The study demonstrates that while the brain successfully enhances the representation of a single instrument during segregation, "listening to the whole" is a more complex and varied cognitive process. The null result for integration suggests that aggregate tracking might not be a simple linear combination of individual instrument envelopes. Furthermore, encoding models that included both instrument envelopes outperformed those using only the aggregate envelope, indicating that the brain maintains distinct representations of individual instruments even when attempting to perceive the ensemble as a whole.

Limitations and Future Directions

The study's focus on non-musicians leaves open the question of whether musical expertise would lead to more consistent integration strategies. Additionally, the steeper envelope of the bassoon likely contributed to its superior tracking; future research should aim for even stricter control of low-level acoustic features. Finally, while EEG provides excellent temporal resolution, future studies using MEG or ECoG could help localize the specific fronto-temporal networks responsible for these attentional shifts.

Summary updated

Abstract

Numerous neuroimaging studies demonstrated that the auditory cortex tracks ongoing speech and that, in multi-speaker environments, tracking of the attended speaker is enhanced compared to the other irrelevant speakers. In contrast to speech, multi-instrument music can be appreciated by attending not only on its individual entities (i.e., segregation) but also on multiple instruments simultaneously (i.e., integration). We investigated the neural correlates of these two modes of music listening using electroencephalography (EEG) and sound envelope tracking. To this end, we presented uniquely composed music pieces played by two instruments, a bassoon and a cello, in combination with a previously validated music auditory scene analysis behavioral paradigm (Disbergen et al., 2018). Similar to results obtained through selective listening tasks for speech, relevant instruments could be reconstructed better than irrelevant ones during the segregation task. A delay-specific analysis showed higher reconstruction for the relevant instrument during a middle-latency window for both the bassoon and cello and during a late window for the bassoon. During the integration task, we did not observe significant attentional modulation when reconstructing the overall music envelope. Subsequent analyses indicated that this null result might be due to the heterogeneous strategies listeners employ during the integration task. Overall, our results suggest that subsequent to a common processing stage, top-down modulations consistently enhance the relevant instrument's representation during an instrument segregation task, whereas such an enhancement is not observed during an instrument integration task. These findings extend previous results from speech tracking to the tracking of multi-instrument music and, furthermore, inform current theories on polyphonic music perception.
BibTeX
@inproceedings{hausfeld2021modulating,
  author = {Lars Gutschalk Hausfeld and Niels R. Disbergen and Giancarlo Valente and Robert J. Zatorre and Elia Formisano},
  title = {{Modulating Cortical Instrument Representations During Auditory Stream Segregation and Integration With Polyphonic Music}},
  booktitle = {Frontiers in Neuroscience},
  year = {2021},
  doi = {10.3389/fnins.2021.635937},
}
Figure