TL;DR
Analyzing musical dynamics (e.g., piano, forte) in singing is notoriously difficult due to a lack of annotated data and the confounding effects of audio mixing and mastering. This paper introduces a semi-automated pipeline to curate 509 aligned vocal performances from the OpenScore Lieder corpus alongside a manually annotated, multi-genre test set. Using a multi-scale CNN with self-attention, the authors demonstrate that psychoacoustic Bark-scale loudness features vastly outperform standard log-Mel spectrograms for frame-level vocal dynamics estimation.
What Problem Does This Paper Address?
In vocal pedagogy and singing voice synthesis (SVS), musical dynamics govern expressive nuance and emotional delivery. Yet within Music Information Retrieval (MIR), dynamics analysis has primarily focused on the piano, leaving vocal dynamics largely understudied.
The primary bottleneck has been data. Annotating dynamics requires painstaking listening, and interpretations are subjective—singers rarely follow score markings literally, and score notations denote relative musical intent rather than absolute physical loudness. Furthermore, commercial recordings undergo compression, equalization, and mixing, which distorts raw energy levels. Consequently, simple signal energy is an unreliable proxy for whether a vocalist is singing gently, powerfully, or shouting.
Methods
The authors tackled both the data bottleneck and model architecture through a multi-stage approach:
[YouTube Recordings]
│
▼
[Vocal Separation] (Demucs v2)
│
▼
[Score-Audio Alignment] (DTW on MusicXML)
│
▼
[Visual Alignment Validation] (CREPE f0 vs. Score Notes)
│
▼
[Dynamics Classification] (Multi-Scale CNN + Self-Attention)
- Semi-Automated Data Curation:
- Source Separation: Isolated vocal tracks from piano-accompanied recordings using Demucs v2.
- Score-to-Audio Alignment: Aligned batch-converted MusicXML scores to extracted vocal tracks using Dynamic Time Warping (DTW).
- Visual Quality Control: Filtered out poor alignments by visually inspecting fundamental frequency contours ($f_0$ from CREPE) against score-note boundaries.
- Label Extraction: Extracted continuous dynamic markings using
music21, assuming consecutive notes retain dynamic markings unless modified, yielding a 10-class scale from pianissississimo ($pppp$) to fortissississimo ($ffff$). - Feature Extraction & Dynamics Modeling:
- Acoustic Features Compared: Evaluated standard log-Mel spectrograms against specific loudness across 240 Bark critical bands computed via Zwicker’s time-varying loudness standard (ISO 532-1:2017) via the MoSQITo library.
- Model Architecture: A multi-scale Convolutional Neural Network (CNN) with a multi-head self-attention mechanism trained with cross-entropy loss to predict dynamics classes per frame.
- Context Variations: Evaluated short (4,096 frames, ~16–17 ms resolution) and long (10,000 frames, ~29–30 ms resolution) temporal contexts to assess how temporal scope affects performance.
Datasets
- Training Set (Lieder Dynamics Dataset): 509 singing performances across 163 MusicXML scores (~25.9 hours of audio) from 25 Romantic-era composers (e.g., Schubert, Schumann, Brahms). On average, each song has 3.12 different recorded interpretations.
- Test Set (Cross-Genre Ground Truth): 25 performances (~1.6 hours) spanning pop, rock, jazz, soul, R&B, and musical theatre. A professional classical vocalist created the reference scores and annotated dynamic markings using a structured, four-pass iterative listening protocol.
Key Insights and Ideas
- Bark Critical Bands Mirror Perceptual Effort: Zwicker loudness in Bark critical bands captures psychoacoustic human loudness perception and timbral changes under vocal strain far better than log-Mel energy distributions, making it robust against post-production loudness normalization.
- Dynamics Are Relative, Demanding Broader Context: Larger temporal receptive fields substantially improved estimation accuracy. Dynamics are contextual transitions rather than absolute decibel thresholds; capturing longer musical phrases helps the self-attention layer ground local frames against the piece's dynamic baseline.
- Upper vs. Lower Dynamic Separation: Relaxed accuracy metrics ($\text{Acc} \pm 1$ and $\text{Acc} \pm 2$) demonstrated that the network rarely confused extreme dynamic ends—for instance, a fortissimo was almost never misclassified as a piano.
Key Results
Bark-scale features systematically outperformed log-Mel representations across both context lengths and all evaluation metrics:
| Perceptual Feature | Sequence Length | Temporal Resolution | Exact Acc (%) | Acc ($\pm 1$) (%) | Acc ($\pm 2$) (%) |
|---|---|---|---|---|---|
| log-Mel | 4,096 | 17.4 ms | 6.95 | 38.46 | 63.02 |
| log-Mel | 10,000 | 29.0 ms | 11.35 | 42.55 | 68.38 |
| Bark | 4,096 | 16.0 ms | 20.44 | 59.17 | 82.24 |
| Bark | 10,000 | 30.0 ms | 20.96 | 60.71 | 84.78 |
- Exact classification accuracy reached 20.96% with Bark features at the 10,000-frame context, nearly double the best log-Mel configuration (11.35%).
- Allowing a 2-class margin of error ($\text{Acc} \pm 2$), the Bark model correctly bounded the dynamic region 84.78% of the time.
Limitations and Future Work
- Absolute vs. Relative Modeling: The task is currently formulated as an absolute 10-class classification problem, whereas musical dynamics are inherently relational within a piece.
- Dataset Class Imbalance: The training set skews heavily toward piano (p), while popular music in the test set clusters around mezzo-piano (mp) and mezzo-forte (mf), limiting cross-genre generalization.
- Complex Expression Markings: Nuanced score directions like calando, poco dolce, or sempre piano cannot be parsed automatically via standard dynamic tags and require customized handling.
- Future Directions: The authors plan to combine pitch contours ($f_0$) with loudness representations, test loss formulations that handle ordinal dynamics classes, and expand the dataset to encompass more composers and non-classical vocal styles.
Summary updated