Music Interpretation and Emotion Perception: A Computational and Neurophysiological Investigation

Vassilis Lyberatos, Spyridon Kantarelis, Ioanna Zioga, Christina Anagnostopoulou, Giorgos Stamou, Anastasia Georgaki

TL;DR

This paper explores how musical expressivity impacts emotion perception by combining computational audio analysis, listener/performer emotion tagging, and neurophysiological (EEG) data. They found that expressive and improvisational performances lead to stronger emotional responses and better performer-audience alignment, suggesting expressivity is key for emotional engagement in music.

What Problem Does This Paper Address?

While we understand that music evokes emotions, the precise mechanisms by which performers convey these emotions and how listeners perceive them are complex. Traditional music analysis often focuses on the score, but the nuances of performance—like timing, dynamics, and articulation—significantly shape listener perception. This study aims to bridge the gap between the performer's intent, the acoustic characteristics of their performance, and the listener's emotional and neural responses, particularly investigating the role of expressivity and improvisation. They are looking to understand how these factors contribute to the perception of and emotional responses to music.

Methods

The researchers employed a multimodal approach, integrating three key data streams:

  • Audio Analysis: They extracted 40 acoustic features (dynamics, rhythm, timbre, articulation) from the music using the Essentia library. These were analyzed across different frequency bands and performance conditions.
  • Explicit Emotional Analysis: Performers and listeners (annotators and audience members) provided emotion tags using the Geneva Emotional Music Scale (GEMS-9) and a "Neutral" option. This allowed for direct comparison of perceived vs. intended emotions.
  • Implicit Emotional Analysis (Neurophysiological): Electroencephalography (EEG) data was collected from performers and audience members. Specifically, they focused on the "alpha power shift" (8-13 Hz) as a measure of neural engagement and resonance.

They then used statistical methods (ANOVA, t-tests, Chi-Square) and interpretable machine learning (Decision Trees) to analyze the relationships between these modalities.

Datasets

The study utilized data collected from 16 recording sessions involving professional musicians and volunteer listeners.

  • Audio Data: Over 200 audio tracks were recorded in WAV format (24-bit PCM, 48 kHz).
  • Annotation Data: Approximately 1,905 emotion tags were collected from audience members and performers.
  • Biosignal Data: EEG data was recorded from performers and audience members.

No new benchmark datasets were introduced; rather, they collected novel data within their experimental setup.

Key Insights and Ideas

  • Expressivity Matters: Expressive performances, characterized by sharper attacks, more frequent notes, and greater spectral variation, lead to stronger emotional responses and better performer-audience alignment.
  • Improvisation Fosters Connection: Improvisational performances, both free and over a drone, showed the highest agreement between performer intentions and listener perceptions, suggesting greater emotional clarity.
  • Different Roles for Acoustic Features: While performers' perception of emotion is influenced by spectral features and their own neurophysiological state (alpha shift), audience perception is more complex, relying on a broader range of acoustic factors like pulse clarity and spectral complexity.
  • Neural Resonance: There was a trend for higher alpha power during improvisation, potentially linked to creativity and the suppression of conventional ideas. While differences in alpha power shifts between performers and audience were observed across conditions, they were not statistically significant.
  • Lift Metric Reveals Emotion-Expressivity Link: The "Lift" metric showed that high-arousal emotions like Power and Tension have a stronger association with expressive performances, while Neutral has a weaker association.

Key Results

  • Acoustic Features: Improvisation showed sharper attack slopes. Repertoire performances had higher onset rates and spectral flux. Expressive performances consistently showed higher attack slope, onset rate, and spectral flux compared to non-expressive ones.
  • Explicit Emotions: The most frequent emotion tags were Peacefulness, Sadness, Joy, Nostalgia, and Tension. Performers frequently tagged "Neutral" more than listeners.
  • Performer-Audience Agreement: High agreement between performer intentions and audience perceptions was observed in Expressive and Improvisation conditions, while Modes and Non-Expressive Repertoire showed significant disagreement.
  • Machine Learning Accuracy: Performer Decision Tree models achieved 89.8% accuracy, and audience models reached 85.9% accuracy in predicting emotion tags.

Limitations and Future Work

  • Limitations: The study acknowledges that the observed differences in alpha rhythm power shifts between performers and audience were not statistically significant. The analysis of EEG data was limited to occipital and temporal channels.
  • Future Work: The authors suggest expanding research to include other biosignal measures, explore different musical genres, and investigate the impact of performers' musical backgrounds. They also propose using advanced machine learning techniques to better model the complex interactions between performance features and emotional responses. Further analysis of specific musical features like notes, key signatures, and harmonic structures is also recommended.

Summary updated

Abstract

This study investigates emotional expression and perception in music performance using computational and neurophysiological methods. The influence of different performance settings, such as repertoire, diatonic modal etudes, and improvisation, as well as levels of expressiveness, on performers' emotional communication and listeners' reactions is explored. Professional musicians performed various tasks, and emotional annotations were provided by both performers and the audience. Audio analysis revealed that expressive and improvisational performances exhibited unique acoustic features, while emotion analysis showed stronger emotional responses. Neurophysiological measurements indicated greater relaxation in improvisational performances. This multimodal study highlights the significance of expressivity in enhancing emotional communication and audience engagement.
BibTeX
@misc{lyberatos2025music,
  author = {Vassilis Lyberatos and Spyridon Kantarelis and Ioanna Zioga and Christina Anagnostopoulou and Giorgos Stamou and Anastasia Georgaki},
  title = {{Music Interpretation and Emotion Perception: A Computational and Neurophysiological Investigation}},
  year = {2025},
  eprint = {2506.01982},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2506.01982},
}
Figure