Pitch-and-Spectrum-Aware Singing Quality Assessment with Bias Correction and Model Fusion

Yu-Fei Shi, Yang Ai, Ye-Xin Lu, Hui-Peng Du, Zhen-Hua Ling

Abstract

Automatic assessment of singing quality is a complex task that requires capturing melodic stability and spectral nuances often overlooked by standard speech models. We introduce PS-SQA, a Pitch-and-Spectrum-aware Singing Quality Assessment framework. By incorporating pitch histograms, neural audio codec features, and a novel bias-correction branch, our system addresses the limitations of data imbalance and conservative score prediction. PS-SQA achieved the highest system-level Spearman Rank Correlation Coefficient (SRCC) in the VoiceMOS Challenge 2024, setting a new benchmark for synthetic singing evaluation.

Introduction

While automatic Mean Opinion Score (MOS) prediction for synthesized speech has matured, evaluating synthesized singing remains a significant challenge. Singing involves intricate melodic structures, precise pitch control, and specific spectral characteristics. Standard self-supervised learning (SSL) models, primarily trained on speech, often fail to capture these musical nuances. Furthermore, singing quality datasets are typically imbalanced, with a scarcity of low-quality samples. This leads to a "conservative" prediction bias where models tend to output scores in the middle range, failing to accurately identify exceptionally high or low-quality voices.

Methods

The PS-SQA framework enhances standard SSL models (e.g., Wav2Vec 2.0, HuBERT) through three primary architectural innovations:

  1. Pitch-Aware Predictors: Rather than using raw pitch values, we utilize Pitch Histograms. By mapping pitch values into a single octave across 120 bins, the model captures melodic stability. Sharp peaks in the histogram indicate consistent note-hitting, a key indicator of singing quality.
  2. Spectrum-Aware Predictors: We integrate the encoder of APCodec, a non-quantized neural audio codec, followed by a Conformer block. This branch focuses on the spectral "texture" and acoustic features, capturing nuances in vocal quality that magnitude-only spectra might miss.
  3. Bias Correction Branch: To mitigate data imbalance, we replace the final linear layer with a three-branch structure: an addition branch for high-quality samples, a subtraction branch for low-quality samples, and a middle branch for average samples.
Architectural variants of the PS-SQA predictors: (a) compressed-pitch-aware, (b) pitch-histogram-aware, and (c) spectrum-aware models.

Finally, the top five performing predictors are fused into an ensemble to produce the final MOS.

Experimental Setup

The system was evaluated using the SingMOS dataset from the VoiceMOS Challenge 2024. The dataset contains 3,189 Mandarin and Japanese singing samples from various sources, including Singing Voice Synthesis (SVS), Singing Voice Conversion (SVC), and natural recordings. The evaluation set included "unseen" systems to test the model's generalization capabilities.

Key Insights and Results

A critical finding of this work is the superiority of pitch histograms over raw pitch sequences. Histograms serve as a "signature" of melodic stability. High-quality singing produces distinct, sharp peaks, whereas poor singing results in a blurred distribution.

Pitch distribution histograms for (a) high-quality and (b) low-quality singing samples.

The bias correction branch significantly improved performance in low-resource score intervals. In the 1.75 to 2.0 MOS range, it reduced Mean Squared Error (MSE) by over 1.0. Overall, PS-SQA achieved a system-level SRCC of 0.888 on the SingMOS test set, outperforming the challenge baseline (0.859) and securing first place in Track 2 of the VoiceMOS Challenge 2024.

Conclusion

PS-SQA demonstrates that integrating domain-specific features like pitch histograms and spectral codec information, combined with targeted bias correction, can significantly improve automatic singing quality assessment. Future work will explore additional predictors for rhythmic accuracy and lyrical clarity to further refine the model's evaluative capabilities.

Summary updated

BibTeX
@inproceedings{shi2024pitch,
  author = {Yu-Fei Shi and Yang Ai and Ye-Xin Lu and Hui-Peng Du and Zhen-Hua Ling},
  title = {{Pitch-and-Spectrum-Aware Singing Quality Assessment with Bias Correction and Model Fusion}},
  booktitle = {SLT},
  year = {2024},
  doi = {10.1109/slt61566.2024.10832260},
}
Figure