Multi-pitch estimation by a joint 2-d representation of pitch and pitch dynamics

Tianyu T. Wang, Thomas F. QuatieriInterspeech 2010

TL;DR

This paper introduces a novel method for estimating multiple fundamental frequencies (pitches) in speech, particularly when those pitches are very close or cross. By jointly analyzing pitch and its rate of change using a 2D representation, their approach significantly improves accuracy in challenging scenarios compared to methods that only consider pitch.

What Problem Does This Paper Address?

Estimating the fundamental frequency (pitch) of speech is crucial for many audio processing tasks, like speech recognition and synthesis. However, when multiple people speak simultaneously (co-channel speech), their pitches can overlap, making it difficult to disentangle them. This problem becomes even harder when the pitches are very close in value or even cross each other over time. Existing methods often struggle in these "close pitch" scenarios, leading to inaccurate estimations or merged tracks. This paper aims to tackle this challenge by developing a more robust multi-pitch estimation technique.

Methods

The core of the proposed method lies in a novel 2D representation of speech signals called the Grating Compression Transform (GCT). The GCT analyzes localized regions of a log-spectrogram, not just for pitch but also for its derivative (rate of change). This joint representation captures the "pitch dynamics" of each speaker.

The algorithm proceeds in several steps:

  1. Short-time Analysis: The speech signal is converted into a log-spectrogram using the Short-Time Fourier Transform (STFT).
  2. GCT Analysis: The GCT is applied to localized regions of the log-spectrogram. From the GCT, seven features are extracted, including estimates of pitch, pitch derivative, and other spectral characteristics.
  3. Linear Discriminant Analysis (LDA)-based Pruning: A data-driven classifier (LDA) is trained to identify and discard "spurious" pitch candidates that are likely to be erroneous due to low signal amplitude or speaker overlap.
  4. Clustering and Kalman Filtering Framework: The pruned candidates are then clustered using k-means to obtain local pitch and pitch-derivative estimates for each speaker. These estimates are fed into a pair of Kalman filters, which use a state-space model to track both pitch and pitch-derivative information for each speaker. This allows for robust pitch tracking and assignment, even when pitches are close.

A reference method ("fo_only") is also used for comparison, which only utilizes pitch information and not the pitch derivative.

Datasets

The evaluation uses a dataset of all-voiced speech mixtures. This dataset was created by mixing recordings of 8 male and 8 female speakers, each speaking 8 different all-voiced utterances. The data was sampled at 16 kHz. For training the LDA-based pruning and Kalman filters, mixtures generated from the first 4 male and 4 female speakers speaking the first 4 sentences were used. The test set consisted of mixtures generated from the remaining speakers and sentences. The test set was further divided into mixtures with "separate" pitch tracks and "close" pitch track conditions, with the latter including crossings and mergings.

Key Insights and Ideas

  • Pitch Dynamics Matter: The most significant insight is that jointly considering pitch and its rate of change (pitch dynamics) provides crucial information for disentangling closely spaced or crossing pitch tracks. This is a key advantage over methods that only focus on instantaneous pitch.
  • GCT as a Powerful Representation: The Grating Compression Transform (GCT) proves effective in capturing both pitch and pitch-derivative information in a unified 2D framework, making it well-suited for multi-pitch analysis.
  • Pruning is Essential: Identifying and removing erroneous pitch candidates early in the process (using LDA) is vital for the subsequent clustering and tracking stages to perform accurately.
  • Kalman Filtering for Robust Tracking: The Kalman filter framework, by modeling both pitch and its derivative, provides a robust mechanism for tracking individual pitch trajectories and assigning them to the correct speaker, even in challenging overlap scenarios.

Key Results

The proposed method, "fo_dfo/dt," significantly outperforms the "fo_only" reference method in "close pitch" conditions.

  • In separate pitch conditions, "fo_dfo/dt" achieved an average RMSE of 12.56 Hz, while "fo_only" had an RMSE of 12.68 Hz, showing comparable performance.
  • In crossing pitch conditions, "fo_dfo/dt" achieved an average RMSE of 13.21 Hz, while "fo_only" had a much higher RMSE of 26.70 Hz. This demonstrates a substantial improvement (nearly half the error) in challenging scenarios.
  • Qualitative results (Figures 5-7) visually confirm that "fo_dfo/dt" is better at tracking individual pitches during crossings and mergings, producing more accurate trajectories.

Limitations and Future Work

  • Similar Pitch and Pitch-Derivatives: A limitation identified is when pitch tracks not only have close values but also very similar pitch derivatives. In such cases, even the proposed method can produce erroneous estimates, as seen near 150 ms in Figure 7.
  • Future Work:
    • The authors plan to explore tracking amplitude fluctuations in the GCT to potentially address scenarios where pitch and pitch-derivative are similar.
  • They also intend to incorporate their method into a system that can handle both voiced and unvoiced speech mixtures, which is a more general and challenging problem.

Summary updated

BibTeX
@inproceedings{wang2010multi,
  author = {Tianyu T. Wang and Thomas F. Quatieri},
  title = {{Multi-pitch estimation by a joint 2-d representation of pitch and pitch dynamics}},
  booktitle = {Interspeech},
  year = {2010},
}
Figure