Automatic Estimation of Singing Voice Musical Dynamics

Jyoti Narang, Nazif Can Tamer, Viviana De La Vega, Xavier Serra

Introduction

In musical performance, "dynamics" refers to the expressive variation in loudness between notes and phrases, typically denoted by markings such as piano (soft) or forte (loud). While computational tools for tracking pitch and tempo have reached a high level of maturity, the automatic estimation of dynamics—especially in singing—remains a significant challenge. This research presents a novel methodology and dataset designed to bridge this gap, moving beyond simple volume measurements to capture the nuanced timbral changes associated with vocal effort.

The Challenges of Dynamics Estimation

Analyzing dynamics in recorded singing is complicated by three primary factors:

  1. Data Scarcity: There is a lack of large-scale, publicly available datasets that pair high-quality audio with ground-truth dynamic annotations.
  2. Subjectivity and Relativity: Dynamics are inherently relative. A mezzo-forte for one singer may be a forte for another, and the interpretation of these markings often depends on the musical context of the surrounding phrases.
  3. The "Mixing" Problem: In professional recordings, the raw audio intensity is frequently altered during the mixing and mastering process. Compression and gain adjustments can make a soft vocal performance sound loud in the final mix, making it difficult for models to distinguish between a singer's actual physical effort and post-production volume adjustments.

Methodology: A Semi-Automatic Pipeline

To address the lack of data, the researchers developed a semi-automatic pipeline to curate a large-scale training set from the OpenScore Lieder corpus (19th-century German art songs):

  • Source Separation: The researchers utilized Demucs v2 to isolate vocal tracks from YouTube recordings, mitigating interference from accompaniment.
  • Automatic Alignment: Using Dynamic Time Warping (DTW), the audio was aligned to digital MusicXML scores. This allowed the researchers to map score-based dynamic markings (e.g., p, mf, f) directly to specific timestamps in the audio.
  • Manual Validation: To ensure data quality, a visualization tool was used to compare extracted pitch (f0) against the score, allowing the team to discard performances with poor alignment.

This process resulted in a training set of 509 performances across 163 scores, totaling approximately 26 hours of audio.

Model Architecture and Feature Selection

The estimation model employs a multi-scale Convolutional Neural Network (CNN) with self-attention. This architecture is designed to process different temporal windows simultaneously, capturing both the immediate characteristics of vocal attacks and the broader context of musical phrasing.

A key contribution of this work is the comparison of input features. The researchers tested standard log-Mel spectrograms against Bark-scale features. The Bark scale is designed to model the specific loudness perception of the human ear across critical frequency bands. The hypothesis was that dynamics are encoded more in the timbre (the distribution of energy across harmonics) than in raw decibel levels.

Experimental Results

The model was evaluated on a high-quality, manually curated cross-genre test set (Pop, Rock, Jazz, R&B) annotated by a professional classical vocalist. The task was to classify audio into 10 dynamic categories ranging from pppp to ffff.

  • Bark-Scale Superiority: Models using Bark-scale features achieved an exact match accuracy of 20.96%, nearly doubling the 11.35% achieved by log-Mel features. This confirms that perceptually-motivated features are far more effective at capturing vocal effort.
  • Relaxed Accuracy: When the evaluation criteria were relaxed to allow for a one-step margin of error (e.g., predicting mf instead of f), the Bark-scale model reached 60.71% accuracy.
  • Broad Differentiation: The model proved highly effective at distinguishing general "loud" vs. "soft" regions, achieving 84.78% accuracy within a two-step margin.

Key Insights and Future Work

The primary insight of this research is that timbre is a more reliable indicator of dynamics than volume. When a singer moves from piano to forte, the physical characteristics of the voice change—becoming richer in harmonics or more "pressed." By focusing on these timbral cues, the model can effectively ignore the artificial volume adjustments introduced during audio mastering.

Future work will focus on integrating pitch features, as vocal effort and pitch are often physically linked, and improving the model's ability to handle relative dynamics like crescendo and diminuendo. Additionally, expanding the dataset to include more diverse genres and composers will improve the generalizability of the system.

Summary updated

BibTeX
@inproceedings{narang2024automatic,
  author = {Jyoti Narang and Nazif Can Tamer and Viviana De La Vega and Xavier Serra},
  title = {{Automatic Estimation of Singing Voice Musical Dynamics}},
  booktitle = {ISMIR},
  year = {2024},
  url = {https://openreview.net/forum?id=nE0iHIPYKD},
}
Figure