Singing voice detection in polyphonic music using predominant pitch

Vishweshwara Rao, S. Ramakrishnan, Preeti RaoInterspeech 2009

TL;DR

This paper introduces novel energy-based features that leverage the inherent pitch instability of the human voice to distinguish it from stable-pitched instruments in polyphonic music. These features, particularly the Sinusoidal Track Harmonic Energy (STHE), demonstrate superior performance in singing voice detection, especially in challenging North Indian Classical Music (NICM) contexts, and also aid in automatically segmenting audio for improved classification.

What Problem Does This Paper Address?

Identifying singing voice within complex musical pieces, known as Singing Voice Detection (SVD), is a crucial task for many Music Information Retrieval (MIR) applications like melody extraction and voice separation. While traditional methods often rely on spectral features to capture timbre, the human voice has a vast and diverse timbral space that can be difficult to represent comprehensively. Furthermore, loud, pitched instruments can easily be misclassified as vocals. This paper tackles the challenge of robust SVD, particularly in non-Western music genres like North Indian Classical Music (NICM), where traditional Western-centric approaches may not be optimal. The authors aim to improve SVD accuracy by focusing on the energy of the singing voice, specifically by exploiting its unique temporal characteristics, and by developing methods to mitigate interference from accompanying instruments.

Methods

The core of this paper lies in two novel pitch-based features:

  • Normalized Harmonic Energy (NHE): This feature quantifies the energy concentrated around the predominant pitch of the singing voice. It sums the energy of harmonics within a certain frequency range (up to 5 kHz) and normalizes it by the maximum energy observed in the entire performance. The idea is that the singing voice, being the dominant sound source, will have significant energy at its fundamental frequency and its harmonics.
  • Sinusoidal Track Harmonic Energy (STHE): This is an advancement over NHE, designed to address the issue of loud, stable-pitched instruments. STHE leverages the temporal instability of vocal pitch (e.g., vibrato, natural pitch fluctuations) compared to the stable pitch contours of instruments like the harmonium. It achieves this by:
  • Predominant Pitch-Based Partial Tracking: Using a melody extraction algorithm to identify the predominant pitch and then tracking sinusoidal components that are likely to be harmonics of this pitch.
  • Standard Deviation Based Track Pruning: This is the key innovation. It prunes sinusoidal tracks that exhibit very little variation in their frequency over time (low standard deviation), effectively removing tracks belonging to stable-pitched instruments. The pruning is done on short, non-overlapping segments to avoid discarding parts of vocal tracks that might temporarily align with instrument harmonics. The energy of the surviving sinusoids (presumably from the singing voice) is then computed and normalized.

These features are then fed into a Gaussian Mixture Model (GMM) classifier for frame-level SVD. Additionally, the STHE feature is used in an audio segmentation framework (based on Foote's novelty detection algorithm) to automatically detect segment boundaries, which are then used to smooth the frame-level classification decisions.

Datasets

  • Training Data: Excerpts from the start, middle, and end of seven polyphonic NICM vocal performances (23 minutes total), featuring male and female singers, accompanied by drone, percussion, and a secondary melodic instrument (SMI) like the harmonium.
  • Testing Dataset 1: Two songs similar to the training data, sung by male and female performers.
  • Testing Dataset 2: Multi-track data from live NICM performances, down-mixed to create a low signal-to-accompaniment ratio (5 dB) for the voice relative to the SMI (harmonium). This dataset is designed to challenge the system with loud instrumental accompaniment.

Ground truth vocal and instrumental frames were manually marked for both training and testing data.

Key Insights and Ideas

  • Pitch Instability as a Discriminator: The paper highlights that the natural pitch fluctuations and expressivity of the human voice (vibrato, jitter, flutter) are significant distinguishing features from the often rigidly stable pitches of instruments. This is a clever shift from solely relying on timbre.
  • Energy Around Predominant Pitch: Focusing on the energy concentrated at the fundamental and its harmonics, guided by the predominant pitch, proves to be a strong indicator of vocal presence, especially when instruments are quieter.
  • Robustness to Loud Instruments: The STHE feature's ability to prune stable instrument tracks is crucial. It allows for accurate SVD even when instruments like the harmonium are as loud as, or louder than, the singing voice.
  • Automatic Boundary Detection for Smoothing: Using STHE to detect audio novelty and segment the audio provides an automatic way to smooth frame-level SVD decisions, improving overall accuracy without manual segmentation.
  • NICM Specific Challenges: The paper implicitly addresses the nuances of NICM, where pitch simultaneity (chords) is less common, and tuning frequencies are not standardized, making Western-centric approaches less effective.

Key Results

  • Frame-Level Classification:
    • Testing Dataset 1: STHE (FS4) achieved a vocal accuracy of 83.45% and an instrumental accuracy of 90.24%. While vocal accuracy was slightly lower than NHE (FS3), instrumental accuracy was significantly higher, outperforming traditional MFCCs (FS1) and spectral features (FS2).
  • Testing Dataset 2 (Low S/A ratio): STHE (FS4) showed a remarkable instrumental classification accuracy of 86.34%, far superior to other features, and a vocal accuracy of 85.62%. This demonstrates its robustness in challenging conditions.
  • Post-Processing (Boundary Detection):
    • Testing Dataset 1: Post-processing using STHE-detected boundaries significantly improved vocal and instrumental accuracies for all feature sets.
  • Testing Dataset 2: Post-processing led to a significant increase in vocal accuracy for all feature sets, though instrumental accuracy saw a slight drop.

Limitations and Future Work

  • Vocal Recall vs. Instrument Recall: The authors note that while pitch-based energy features improve instrumental recall (correctly identifying non-vocal segments), they might slightly underperform on vocal recall (correctly identifying vocal segments).
  • Voice Not Always Dominant: The paper acknowledges that the singing voice is not always the dominant sound source, which questions the sole reliance on energy-based features. This is particularly relevant in datasets like Testing Dataset 2.
  • Melody Extractor Dependency: The performance of NHE and STHE relies on the accuracy of the predominant pitch estimation from the melody extractor.
  • Future Work: The authors propose investigating a hybrid approach that combines timbral and energy features for both classification and boundary detection, aiming for even better performance. They also mention ongoing work on improving melody extractors for better robustness to co-occurring instruments.

Summary updated

BibTeX
@inproceedings{rao2009singing,
  author = {Vishweshwara Rao and S. Ramakrishnan and Preeti Rao},
  title = {{Singing voice detection in polyphonic music using predominant pitch}},
  booktitle = {Interspeech},
  year = {2009},
}
Figure