Automatic Estimation of Singing Voice Musical Dynamics

Jyoti Narang, Nazif Can Tamer, Viviana De La Vega, Xavier Serra

TL;DR

Analyzing musical dynamics (e.g., piano, forte) in singing is notoriously difficult due to a lack of annotated data and the confounding effects of audio mixing and mastering. This paper introduces a semi-automated pipeline to curate 509 aligned vocal performances from the OpenScore Lieder corpus alongside a manually annotated, multi-genre test set. Using a multi-scale CNN with self-attention, the authors demonstrate that psychoacoustic Bark-scale loudness features vastly outperform standard log-Mel spectrograms for frame-level vocal dynamics estimation.

What Problem Does This Paper Address?

In vocal pedagogy and singing voice synthesis (SVS), musical dynamics govern expressive nuance and emotional delivery. Yet within Music Information Retrieval (MIR), dynamics analysis has primarily focused on the piano, leaving vocal dynamics largely understudied.

The primary bottleneck has been data. Annotating dynamics requires painstaking listening, and interpretations are subjective—singers rarely follow score markings literally, and score notations denote relative musical intent rather than absolute physical loudness. Furthermore, commercial recordings undergo compression, equalization, and mixing, which distorts raw energy levels. Consequently, simple signal energy is an unreliable proxy for whether a vocalist is singing gently, powerfully, or shouting.

Methods

The authors tackled both the data bottleneck and model architecture through a multi-stage approach:

[YouTube Recordings] 
       │
       ▼
 [Vocal Separation] (Demucs v2)
       │
       ▼
[Score-Audio Alignment] (DTW on MusicXML)
       │
       ▼
[Visual Alignment Validation] (CREPE f0 vs. Score Notes)
       │
       ▼
[Dynamics Classification] (Multi-Scale CNN + Self-Attention)
  1. Semi-Automated Data Curation:
  2. Source Separation: Isolated vocal tracks from piano-accompanied recordings using Demucs v2.
  3. Score-to-Audio Alignment: Aligned batch-converted MusicXML scores to extracted vocal tracks using Dynamic Time Warping (DTW).
  4. Visual Quality Control: Filtered out poor alignments by visually inspecting fundamental frequency contours ($f_0$ from CREPE) against score-note boundaries.
  5. Label Extraction: Extracted continuous dynamic markings using music21, assuming consecutive notes retain dynamic markings unless modified, yielding a 10-class scale from pianissississimo ($pppp$) to fortissississimo ($ffff$).
  6. Feature Extraction & Dynamics Modeling:
  7. Acoustic Features Compared: Evaluated standard log-Mel spectrograms against specific loudness across 240 Bark critical bands computed via Zwicker’s time-varying loudness standard (ISO 532-1:2017) via the MoSQITo library.
  8. Model Architecture: A multi-scale Convolutional Neural Network (CNN) with a multi-head self-attention mechanism trained with cross-entropy loss to predict dynamics classes per frame.
  9. Context Variations: Evaluated short (4,096 frames, ~16–17 ms resolution) and long (10,000 frames, ~29–30 ms resolution) temporal contexts to assess how temporal scope affects performance.

Datasets

  • Training Set (Lieder Dynamics Dataset): 509 singing performances across 163 MusicXML scores (~25.9 hours of audio) from 25 Romantic-era composers (e.g., Schubert, Schumann, Brahms). On average, each song has 3.12 different recorded interpretations.
  • Test Set (Cross-Genre Ground Truth): 25 performances (~1.6 hours) spanning pop, rock, jazz, soul, R&B, and musical theatre. A professional classical vocalist created the reference scores and annotated dynamic markings using a structured, four-pass iterative listening protocol.

Key Insights and Ideas

  • Bark Critical Bands Mirror Perceptual Effort: Zwicker loudness in Bark critical bands captures psychoacoustic human loudness perception and timbral changes under vocal strain far better than log-Mel energy distributions, making it robust against post-production loudness normalization.
  • Dynamics Are Relative, Demanding Broader Context: Larger temporal receptive fields substantially improved estimation accuracy. Dynamics are contextual transitions rather than absolute decibel thresholds; capturing longer musical phrases helps the self-attention layer ground local frames against the piece's dynamic baseline.
  • Upper vs. Lower Dynamic Separation: Relaxed accuracy metrics ($\text{Acc} \pm 1$ and $\text{Acc} \pm 2$) demonstrated that the network rarely confused extreme dynamic ends—for instance, a fortissimo was almost never misclassified as a piano.

Key Results

Bark-scale features systematically outperformed log-Mel representations across both context lengths and all evaluation metrics:

Perceptual Feature Sequence Length Temporal Resolution Exact Acc (%) Acc ($\pm 1$) (%) Acc ($\pm 2$) (%)
log-Mel 4,096 17.4 ms 6.95 38.46 63.02
log-Mel 10,000 29.0 ms 11.35 42.55 68.38
Bark 4,096 16.0 ms 20.44 59.17 82.24
Bark 10,000 30.0 ms 20.96 60.71 84.78
  • Exact classification accuracy reached 20.96% with Bark features at the 10,000-frame context, nearly double the best log-Mel configuration (11.35%).
  • Allowing a 2-class margin of error ($\text{Acc} \pm 2$), the Bark model correctly bounded the dynamic region 84.78% of the time.

Limitations and Future Work

  • Absolute vs. Relative Modeling: The task is currently formulated as an absolute 10-class classification problem, whereas musical dynamics are inherently relational within a piece.
  • Dataset Class Imbalance: The training set skews heavily toward piano (p), while popular music in the test set clusters around mezzo-piano (mp) and mezzo-forte (mf), limiting cross-genre generalization.
  • Complex Expression Markings: Nuanced score directions like calando, poco dolce, or sempre piano cannot be parsed automatically via standard dynamic tags and require customized handling.
  • Future Directions: The authors plan to combine pitch contours ($f_0$) with loudness representations, test loss formulations that handle ordinal dynamics classes, and expand the dataset to encompass more composers and non-classical vocal styles.

Summary updated

Abstract

Musical dynamics form a core part of expressive singing voice performances. However, automatic analysis of musical dynamics for singing voice has received limited attention partly due to the scarcity of suitable datasets and a lack of clear evaluation frameworks. To address this challenge, we propose a methodology for dataset curation. Employing the proposed methodology, we compile a dataset comprising 509 musical dynamics annotated singing voice performances, aligned with 163 score files, leveraging state-of-the-art source separation and alignment techniques. The scores are sourced from the OpenScore Lieder corpus of romantic-era compositions, widely known for its wealth of expressive annotations. Utilizing the curated dataset, we train a multi-head attention based CNN model with varying window sizes to evaluate the effectiveness of estimating musical dynamics. We explored two distinct perceptually motivated input representations for the model training: log-Mel spectrum and bark-scale based features. For testing, we manually curate another dataset of 25 musical dynamics annotated performances in collaboration with a professional vocalist. We conclude through our experiments that bark-scale based features outperform log-Mel-features for the task of singing voice dynamics prediction. The dataset along with the code is shared publicly for further research on the topic.
BibTeX
@inproceedings{narang2024automatic,
  author = {Jyoti Narang and Nazif Can Tamer and Viviana De La Vega and Xavier Serra},
  title = {{Automatic Estimation of Singing Voice Musical Dynamics}},
  booktitle = {ISMIR},
  year = {2024},
  url = {https://openreview.net/forum?id=nE0iHIPYKD},
}
Figure