2-d processing of speech for multi-pitch analysis

Tianyu T. Wang, Thomas F. QuatieriInterspeech 2009

TL;DR

This paper introduces a novel 2D processing technique called the Grating Compression Transform (GCT) for analyzing speech with multiple simultaneous pitches. By mapping harmonic structures to a 2D space, their Multi-Region Analysis (MRA) method can effectively separate and estimate individual pitches, outperforming traditional methods that struggle with overlapping harmonic content.

What Problem Does This Paper Address?

Estimating the fundamental frequency (pitch) of speech is crucial for many audio processing tasks. However, a significant challenge arises when multiple speakers or vocal sources produce sound simultaneously, resulting in a complex acoustic signal with overlapping harmonic structures. Traditional methods often rely on analyzing the signal along a single dimension (time or frequency) or using short-time autocorrelation, which struggle to disentangle these concurrent pitches, especially when their harmonic content overlaps or their pitches are close. This paper aims to address this limitation by developing a more robust method for multi-pitch analysis.

Methods

The core of the paper's contribution lies in the Grating Compression Transform (GCT). This technique involves:

  • Spectrogram Analysis: First, a standard short-time Fourier transform (STFT) is computed to create a spectrogram, representing the signal's frequency content over time.
  • 2D Fourier Transform on Local Regions: Instead of analyzing the spectrogram directly, the authors apply a 2D Fourier transform to localized, narrowband regions of the spectrogram's magnitude (or log-magnitude). This transforms the harmonic line structures within these regions into concentrated entities in a new 2D space.
  • Multi-Region Analysis (MRA): The key innovation is Multi-Region Analysis (MRA). The GCT of localized time-frequency regions is analyzed to extract pitch candidates. These candidates are then combined across multiple regions to obtain separate pitch estimates for each speech-signal component at a given point in time. This approach explicitly accounts for pitch dynamics within localized segments.
  • Improved Pitch Estimation Metric: The authors found that a specific component of the GCT, denoted as @cos θ, better represents pitch across all frequency regions compared to previous methods that relied on @ (radial distance). This is because @cos θ is inversely related to the vertical spacing between harmonic peaks in the spectrogram.

Datasets

The paper uses both synthetic and real speech mixtures for evaluation.

  • Synthetic Speech: Synthesized concurrent vowels with linear pitch trajectories were created using a glottal pulse train and an all-pole formant envelope.
  • Real Speech: Two all-voiced sentences spoken by a male and female were used, mixed to create scenarios with separate and crossing pitch trajectories.

The paper does not introduce a new benchmark dataset but rather uses these mixtures to evaluate their proposed method against existing approaches.

Key Insights and Ideas

  • 2D Representation for Pitch Separability: The GCT's ability to map harmonic structures into a 2D space is fundamental. This transformation allows for the separation of pitch information that would be indistinguishable in a 1D representation.
  • Multi-Region Analysis (MRA) is Key: Analyzing multiple localized regions of the spectrogram, rather than just one, is crucial for robust multi-pitch estimation. This allows the system to leverage different harmonic structures and dynamics across the frequency spectrum.
  • Pitch Dynamics are Explicitly Captured: The GCT's representation inherently captures the temporal dynamics of pitch trajectories, which is a significant advantage over autocorrelation methods that primarily focus on instantaneous pitch.
  • @cos θ as a Superior Pitch Metric: The discovery that @cos θ is a more reliable indicator of pitch across different frequency bands is a significant refinement, improving the accuracy of the GCT-based analysis.

Key Results

  • Improved Accuracy: The GCT-based MRA method demonstrated significantly lower average percentage errors (%error avg) compared to single-region analysis for both synthetic and real speech mixtures.
  • For synthetic speech, the "oracle" (ideal) estimates achieved %error avg < 0.04%, and the clustering-based MRA achieved %error avg < 1.75%.
  • For real speech, the oracle estimates achieved 0.00% average error.
  • Separability of Similar Pitches: The GCT was shown to successfully separate pitch information from two speakers with similar energies and pitch values by exploiting their temporal dynamics.
  • Outperforms Single-Region Analysis: In all tested cases, both the oracle and clustering of GCT-based MRA pitch candidates outperformed the single-region analysis method, highlighting the benefit of using multiple regions.

Limitations and Future Work

  • Clustering Limitations: For real speech with crossing pitch trajectories, the median-based clustering method sometimes struggled to assign pitch values correctly, especially when one speaker's energy dominated. This suggests that the post-processing (clustering) could be improved to better account for energy variations.
  • Future Work: The authors plan to develop a full multi-pitch tracking system using this framework and evaluate it on a larger dataset. They also aim to improve the post-processing methods for more robust pitch assignment.

Summary updated

BibTeX
@inproceedings{wang2009d,
  author = {Tianyu T. Wang and Thomas F. Quatieri},
  title = {{2-d processing of speech for multi-pitch analysis}},
  booktitle = {Interspeech},
  year = {2009},
}
Figure