Interpretable Singing Emotion Recognition Using Acoustic and Vocal Technique Features

Fotini Kiriakopoulou, Vassilis Lyberatos, Spyridon Kantarelis, Athanasios Voulodimos, Giorgos StamouarXiv 2026 · arXiv

TL;DR

Researchers have developed a new framework for Singing Emotion Recognition (SiER) that moves away from "black box" deep learning toward human-interpretable vocal features. By incorporating singing-specific descriptors like vibrato and pharyngeal resonance, the model not only achieves high accuracy but also reveals how specific vocal techniques—such as the interaction between vibrato and articulation rate—drive emotional perception.

What Problem Does This Paper Address?

Most current emotion recognition systems rely on deep learning models that act as "black boxes." While they perform well, they offer little insight into why a specific audio clip sounds "sad" or "happy." Furthermore, these models often treat singing as just another form of speech, ignoring the unique acoustic properties of the singing voice, such as sustained harmonics and deliberate vocal techniques.

In applications like music recommendation or performance analysis, it is crucial to understand the underlying acoustic cues. This paper aims to bridge the gap between high-performance machine learning and human-interpretable musicology by identifying a compact set of singing-specific features that remain stable across different singers.

Methods

The authors proposed a three-stage framework designed for both performance and transparency:

  1. Feature Extraction: They extracted 130 scalar features categorized into four dimensions:
  2. Prosodic: Pitch ($f_0$) and intensity.
  3. Spectral: Timbre-related cues like the alpha ratio and formant frequencies.
  4. Phonetic: Articulation rate and phoneme duration.
  5. Vocal Technique: A novel addition using a custom dual-input CNN to detect six techniques: vibrato, glissando, mix, pharyngeal, falsetto, and breathy.
  6. Feature Selection: To ensure the model generalizes well, they used a nested Leave-One-Singer-Out (LOSO) framework. This identifies features that are consistently useful across different vocalists rather than those that just happen to fit a specific person's voice.
  7. Interpretability Analysis: The team employed SHAP (SHapley Additive exPlanations) to quantify feature importance and GRAD-CAM to visualize which parts of a spectrogram the vocal technique extractor was actually "looking at."
Fig. 1. Overview of the proposed framework.

Datasets

The study utilized three distinct datasets to ensure robustness:

  • RAVDESS: A baseline dataset of English singing samples featuring 23 actors portraying six emotions.
  • GTSinger: A more realistic and diverse dataset including English, French, and German subsets, encompassing both pop and bel canto (opera) styles.
  • VocalSet: Used as an out-of-domain evaluation set to validate the learned vocal technique representations against ground-truth technical labels.

Key Insights and Ideas

  • Singing is not Speech: The "singing-optimized" feature set significantly outperformed standard speech-based features (like those from the eGeMAPS set) in terms of stability across different singers.
  • The Vibrato-Articulation Interaction: One of the most interesting "aha" moments came from the SHAP analysis. The model found that vibrato's emotional meaning is context-dependent: at slow articulation rates, high vibrato variability signals sadness (a "plaintive" delivery), while at fast rates, it signals happiness (a "vibrant" delivery).
  • Pharyngeal Resonance as a Key Cue: Pharyngeal technique emerged as a top predictor for low-valence (sad) states, highlighting how singers manipulate resonance to convey deep emotion.
  • Visualizing Technique: Using GRAD-CAM, the researchers confirmed that their technique extractor was attending to acoustically meaningful patterns, such as the periodic frequency modulations of vibrato, rather than spurious noise.
Fig. 2. GRAD-CAM on VocalSet: (a) vibrato, (b) belt.

Key Results

The singing-optimized feature set demonstrated superior performance and consistency:

  • On RAVDESS: The singing-optimized set achieved a 79.6% Unweighted Average Recall (UAR), outperforming the speech-optimized set (76.2%) while reducing cross-singer variability by approximately 32%.
  • Ablation Studies: As shown in the table below, while prosody is the strongest single predictor, the combination of all features ("All-but-1" tests) provides the most stable results with the lowest standard deviation (SD).
Ablation study results showing that while prosody dominates accuracy, the full feature set provides the best balance of performance and stability.
  • Cross-Corpus Validation: When applied to VocalSet, the model successfully mapped high-arousal emotions (like anger) to dynamic techniques like "belt" and "forte," while mapping sadness to "breathy" and "slow legato" techniques.

Limitations and Future Work

The authors acknowledge that their findings represent statistical associations rather than causal mechanisms. Because the vocal technique features rely on CNN-inferred probabilities, there is a risk of "label noise" if the technique extractor itself makes an error. Furthermore, the datasets used—while diverse—are still limited in the number of singers and the range of emotions (GTSinger only uses "happy" vs. "sad").

Future research aims to expand this framework to more diverse musical genres, languages, and traditional vocal styles to see if these "interpretable vocabularies" of singing hold true globally.

Summary updated

Abstract

Singing emotion recognition relies predominantly on deep learning models that achieve strong performance but offer limited insight into the vocal characteristics driving emotion, of- ten overlooking the singing voice as a distinct signal. We propose an interpretable singing emotion recognition framework using a compact set of singing-specific descriptors across prosody, spec- tral features, phonetics, and vocal technique. To decode model decisions and link predictions to measurable acoustic phenomena, we employ SHAP and apply GRAD-CAM to qualitatively assess the learned representations of vocal techniques. Evaluations show that singing-specific features reduce cross-singer variability, while vocal technique descriptors provide interpretable insights into singing expression. Furthermore, out-of-domain evaluation on VocalSet reveals interesting associations between emotional expression and ground-truth technical execution. The analysis highlights vibrato and pharyngeal techniques as salient cues of emotional delivery, with SHAP indicating that vibrato is context-dependent on articulation rate. These findings emphasize the value of domain-specific features for interpretable singing emotion analysis.
BibTeX
@misc{kiriakopoulou2026interpretable,
  author = {Fotini Kiriakopoulou and Vassilis Lyberatos and Spyridon Kantarelis and Athanasios Voulodimos and Giorgos Stamou},
  title = {{Interpretable Singing Emotion Recognition Using Acoustic and Vocal Technique Features}},
  year = {2026},
  eprint = {2610.32629},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.32629},
}
Figure