Assessing Top-Down and Bottom-Up Contributions to Auditory Stream Segregation and Integration With Polyphonic Music

Niels R. Disbergen, Giancarlo Valente, Elia Formisano, Robert J. ZatorreFrontiers in Neuroscience 2018 · DOI

TL;DR

This paper introduces and validates a novel experimental paradigm using custom-composed polyphonic music to study how our brains segregate and integrate auditory streams, mimicking real-world listening. The paradigm successfully trains non-musicians to perform attention-switching tasks and can be adapted for fMRI studies, paving the way for investigating auditory scene analysis in more complex, naturalistic contexts.

What Problem Does This Paper Address?

Understanding how we make sense of complex auditory environments – like a bustling party or an orchestra – is a fundamental challenge in auditory neuroscience. This involves two key processes: auditory stream segregation (separating individual sound sources) and auditory stream integration (combining related sounds into a coherent whole). While much research has focused on simplified auditory scenes (like pure tones), real-world listening is far richer and more complex, especially with music. Polyphonic music, with its multiple interwoven melodic lines, presents a unique challenge and opportunity: it allows for both segregation (hearing individual instruments) and integration (perceiving the overall harmonic structure). Existing paradigms often lack the ecological validity of naturalistic music, making it difficult to translate findings to real-world listening or neuroimaging studies. This paper aims to bridge this gap by developing and validating a task that uses realistic polyphonic music to investigate the interplay of bottom-up (stimulus-driven) and top-down (attention-driven) processes in auditory scene analysis.

Methods

The core of this research is a novel psychophysical paradigm designed to assess auditory stream segregation and integration using custom-composed two-voice polyphonic music.

  • Stimuli: The music pieces were specifically composed to be recognizable as polyphonic yet allow for controlled manipulation. They featured two voices (synthesized bassoon and cello) and incorporated temporal modulations called "triplets" (three eighth notes in the time of one beat). These triplets served as the target stimuli for participants to detect.
  • Experiment 1 (Attention Modulation): Participants were instructed to attend to either one of the individual instruments (bassoon or cello) or the aggregate (both instruments together). Their task was to detect whether these triplet patterns were present within their attended instrument(s) or across instruments. This allowed researchers to assess how attention influences segregation (attending to one instrument) and integration (attending to both).
  • Experiment 2 (Bottom-Up Manipulation): This experiment built upon Experiment 1 by introducing a bottom-up manipulation: the timbre distance between the two instruments was varied across three levels (maximum, intermediate, and minimum). This allowed researchers to investigate how the physical differences between sound sources (timbre) interact with attentional control.
  • fMRI Adaptation: Experiment 2 was also validated within a functional Magnetic Resonance Imaging (fMRI) environment, comparing performance in a sound-attenuated chamber (LAB group) with performance inside the scanner (SCAN group) to assess the impact of scanner noise.
  • Behavioral Metric: A key innovation is the use of a behavioral performance metric (accuracy in detecting triplets) to validate listener instructions and assess their locus of attention. High accuracy indicated successful segregation or integration.
  • Statistical Analysis: Hierarchical Bayesian models were employed to analyze the data, allowing for robust estimation of parameters and accounting for individual differences and potential ceiling effects in performance.

Datasets

The study utilized custom-composed polyphonic music stimuli.

  • Experimental Stimuli: Twenty polyphonic counterpoint music pieces were created for the main experiments.
  • Training and Testing Stimuli: Sixty-two additional unique pieces were custom-written for training and testing purposes, ensuring that participants learned the task without memorizing specific experimental stimuli.
  • Novel Benchmark: The custom-composed music pieces and the experimental paradigm itself serve as a novel benchmark for studying auditory scene analysis in more ecologically valid musical contexts, differing from traditional, more elementary auditory stimuli.

Key Insights and Ideas

  • Music as a Rich Auditory Scene: Polyphonic music is an excellent naturalistic stimulus for studying auditory scene analysis, offering a complex yet controllable environment for exploring segregation and integration.
  • Attentional Control is Key: Even with complex musical stimuli, participants could be trained to reliably switch their attention between individual instruments or the combined sound, demonstrating the power of top-down control.
  • Timbre Matters, But Not Always Simply: While timbre differences are a crucial bottom-up cue, their effect on performance wasn't straightforward. The study found a main effect of timbre distance, but this was more pronounced in subjects with lower overall performance, suggesting that task difficulty can mask timbre effects in high performers.
  • fMRI Compatibility: The paradigm is robust enough to be used in fMRI environments, with scanner noise not significantly impacting performance, which is a critical step for neuroimaging research in this area.
  • Training is Effective: Non-musicians could be trained to perform the task with high accuracy, highlighting the plasticity of auditory processing and the effectiveness of the training protocol.

Key Results

  • Experiment 1: Participants (N=29, non-musicians) achieved high accuracy in detecting triplets, with no significant group differences across attentive conditions (segregation vs. integration). This indicates successful training and task compliance.
  • Experiment 2:
    • A main effect of instrument timbre distance was observed, suggesting that the degree of timbre difference influences performance.
  • However, within specific attention conditions (e.g., attending to bassoon vs. cello), there were no significant differences in performance related to timbre distance.
  • Correlation analysis revealed that subjects with lower overall performance showed a stronger influence of timbre distance, suggesting that task difficulty can mask timbre effects in high performers.
  • Performance in the fMRI scanner environment (SCAN group) was comparable to that in a sound-attenuated chamber (LAB group), indicating the paradigm's suitability for neuroimaging.

Limitations and Future Work

  • Partial Ceiling Effect: The authors acknowledge a partial "ceiling effect" on scores, meaning many participants performed at near-perfect accuracy. This limits the ability to detect subtle differences between conditions.
  • Future Work: To address this, future iterations could increase task difficulty by, for example, reducing pitch distance between melodic lines or further reducing timbre differences.
  • Specificity of Learning: The observed learning effects might be specific to the trained musical stimuli and instruments, rather than a general gain in stream segregation ability.
  • Future Work: Future studies could investigate whether these effects transfer to other instruments or more common streaming tasks (like speech in noise) to understand the generalizability of music-specific auditory processing.
  • Interaction Effects: While trends were observed, significant interactions between attention and timbre distance were not found, possibly due to the high performance levels.
  • Future Work: Further investigation into these interactions, particularly with more challenging tasks, is warranted.
  • Reaction Time: The study focused on accuracy.
  • Future Work: Incorporating reaction time measures could provide additional insights into cognitive load and processing differences between conditions.
  • Musicianship: The study focused on non-musicians.
  • Future Work: The paradigm could be used to investigate plasticity and training effects in highly trained musicians, exploring how their expertise might influence segregation and integration strategies.

Summary updated

Abstract

Polyphonic music listening well exemplifies processes typically involved in daily auditory scene analysis situations, relying on an interactive interplay between bottom-up and top-down processes. Most studies investigating scene analysis have used elementary auditory scenes, however real-world scene analysis is far more complex. In particular, music, contrary to most other natural auditory scenes, can be perceived by either integrating or, under attentive control, segregating sound streams, often carried by different instruments. One of the prominent bottom-up cues contributing to multi-instrument music perception is their timbre difference. In this work, we introduce and validate a novel paradigm designed to investigate, within naturalistic musical auditory scenes, attentive modulation as well as its interaction with bottom-up processes. Two psychophysical experiments are described, employing custom-composed two-voice polyphonic music pieces within a framework implementing a behavioral performance metric to validate listener instructions requiring either integration or segregation of scene elements. In Experiment 1, the listeners' locus of attention was switched between individual instruments or the aggregate (i.e., both instruments together), via a task requiring the detection of temporal modulations (i.e., triplets) incorporated within or across instruments. Subjects responded post-stimulus whether triplets were present in the to-be-attended instrument(s). Experiment 2 introduced the bottom-up manipulation by adding a three-level morphing of instrument timbre distance to the attentional framework. The task was designed to be used within neuroimaging paradigms; Experiment 2 was additionally validated behaviorally in the functional Magnetic Resonance Imaging (fMRI) environment. Experiment 1 subjects (<i>N</i> = 29, non-musicians) completed the task at high levels of accuracy, showing no group differences between any experimental conditions. Nineteen listeners also participated in Experiment 2, showing a main effect of instrument timbre distance, even though within attention-condition timbre-distance contrasts did not demonstrate any timbre effect. Correlation of overall scores with morph-distance effects, computed by subtracting the largest from the smallest timbre distance scores, showed an influence of general task difficulty on the timbre distance effect. Comparison of laboratory and fMRI data showed scanner noise had no adverse effect on task performance. These Experimental paradigms enable to study both bottom-up and top-down contributions to auditory stream segregation and integration within psychophysical and neuroimaging experiments.
BibTeX
@inproceedings{disbergen2018assessing,
  author = {Niels R. Disbergen and Giancarlo Valente and Elia Formisano and Robert J. Zatorre},
  title = {{Assessing Top-Down and Bottom-Up Contributions to Auditory Stream Segregation and Integration With Polyphonic Music}},
  booktitle = {Frontiers in Neuroscience},
  year = {2018},
  doi = {10.3389/fnins.2018.00121},
}
Figure