A comparison of singing evaluation algorithms

Partha LalInterspeech 2006

TL;DR

This paper presents and compares two algorithms for evaluating how closely a user's singing matches an original recording, focusing on pitch accuracy. They developed a system deployed in a telephone-based entertainment application, finding that a Viterbi alignment of pitch curves provides a robust metric for assessing singing performance.

What Problem Does This Paper Address?

The paper tackles the challenge of automatically evaluating singing performance, a task that has gained popularity with the rise of televised singing competitions. While human judges can assess vocal similarity, creating an automated system that provides a reliable and understandable score is complex. The core problem is to quantify how well a user's rendition of a sung melody matches the original, going beyond simple pitch detection to account for timing and subtle variations. This has potential applications beyond entertainment, such as in language learning.

Methods

The authors explored two main approaches for comparing sung clips:

  • Simple Pitch Comparison: This method involved extracting a fixed number of pitch samples (50) from both the original and user recordings. These samples were then linearly interpolated over unvoiced regions. The Euclidean distance between these pitch vectors served as the primary metric, with a higher distance indicating a greater difference. Parameters like the number of samples and an interpolation threshold were optimized.
  • Dynamically-aligned Pitch Comparison: Recognizing the limitations of the simple approach in handling timing discrepancies, this method employed a Viterbi alignment algorithm. This technique aligns the pitch curves of the user's rendition and the original recording, allowing for substitutions, insertions, and deletions. The algorithm was modified to penalize pitch differences proportionally to their magnitude. This approach also normalized pitch values by the mean and standard deviation of the reference clip.

Additionally, the paper discusses a method to penalize humming by combining the pitch-based score with confidence scores from a speech recognizer.

Datasets

  • Development Corpus: An existing corpus of sung speech was used for initial design and optimization.
  • Evaluation Corpora:
    • A corpus of 50 recordings of "Row, row, row your boat" and 50 recordings of "O Canada" was used for evaluating the simple pitch comparison method. Human judges ranked these renditions based on closeness to the original.
  • A corpus of 35 renditions of four different songs was collected for evaluating the dynamically-aligned pitch comparison. Later, a second corpus of 27 recordings of two different songs, matching the target deployment length of 30 seconds, was also used. Human judges independently ranked these renditions.

Key Insights and Ideas

  • Timing is Crucial: The simple pitch comparison method, while straightforward, was overly sensitive to timing errors. This highlighted the need for alignment algorithms that can handle temporal variations.
  • Viterbi Alignment for Singing: Applying Viterbi alignment, a standard technique in speech recognition, proved effective for matching sung pitch curves. This allowed for a more robust comparison by accounting for slight shifts in timing.
  • Pitch Difference as a Metric: Measuring the total pitch difference after alignment served as a reliable distance metric, which could then be transformed into a user-friendly score out of ten.
  • Humming Detection is a Challenge: The system's reliance on pitch alone meant it struggled to differentiate between singing and humming. Combining pitch information with speech recognition confidence scores was explored as a solution.
  • Repeat Callers Indicate Success: The high rate of repeat callers (73%) for the deployed entertainment application suggests that the scoring system was perceived as plausible and encouraging by users.

Key Results

  • The Dynamically-aligned Pitch Comparison method was found to be superior to the Simple Pitch Comparison.
  • For the simple pitch comparison, the optimal pitch vector length was determined to be 100 samples, and the optimal voiced/unvoiced threshold was 12.5%.
  • The insertion and deletion penalty for the Viterbi alignment was optimized, leading to improved performance.
  • The humming-sensitive score, incorporating speech recognition confidence, showed a relative improvement of 37% in the evaluation metric (mean eval₂).

Limitations and Future Work

  • Humming Detection: The paper acknowledges that the preliminary humming detection method needs further verification on a larger corpus.
  • Source Separation: Ideally, the system would benefit from separate vocal tracks for reference clips, either obtained directly or through source separation techniques.
  • Probabilistic Measures: The authors suggest that replacing the current distance measure in the Viterbi alignment with a probabilistic measure (e.g., negative log probability of pitch difference) could further enhance the system.
  • Real-time Feedback: The paper notes that existing systems offer real-time feedback, which their current system does not provide.

Summary updated

BibTeX
@inproceedings{lal2006comparison,
  author = {Partha Lal},
  title = {{A comparison of singing evaluation algorithms}},
  booktitle = {Interspeech},
  year = {2006},
}
Figure