TL;DR
This paper introduces a novel multipitch tracker (MPT) that excels at segmenting mixed speech into coherent fragments belonging to individual speakers. By systematically modeling pitch doubling and halving errors and emphasizing fragment coherence, the MPT improves downstream speech recognition performance, even in challenging noisy conditions.
What Problem Does This Paper Address?
The paper tackles the difficult problem of separating and segmenting speech from multiple overlapping sources when only a single audio channel (monaural recording) is available. Imagine trying to understand a conversation in a crowded room – it's hard to isolate individual voices. This is a crucial challenge for applications like automatic speech recognition (ASR) and speaker diarization, where identifying who said what and when is essential. Traditional methods often struggle when sources have similar pitches or when the fundamental frequency is weak or absent, leading to inaccurate segmentation and poor recognition. This work aims to create more reliable, "coherent" speech fragments that can be confidently assigned to a single speaker.
Methods
The core of the proposed system is the Multipitch Tracker (MPT). Here's a breakdown of its key components:
- Pitch Detection: An autocorrelogram (ACG) based algorithm is used to detect multiple pitch candidates within the mixed speech signal. This involves processing the signal through a filter bank and analyzing the resulting autocorrelations. A "summary ACG" (sACG) is computed from low-frequency channels, and peaks exceeding a threshold are considered potential pitch candidates.
- Pitch Dynamics Modeling: The MPT uses a Hidden Markov Model (HMM) framework to track these pitch candidates over time. A novel aspect is the explicit modeling of pitch doubling and halving errors. This is crucial because these errors can lead to incorrect pitch assignments, especially when the fundamental frequency is weak. The system models these dynamics using Gaussian Mixture Models (GMMs) trained on clean speech data, accounting for how pitch changes frame-by-frame.
- Distractor Modeling: To handle spurious peaks or noise, an independent noise model is employed. This model captures observations that are not attributed to actual speech sources.
- Fragment Generation: Once the MPT identifies coherent voiced segments (regions of continuous voicing), these segments are transformed into spectro-temporal fragments. A "soft decision" mechanism is used to assign dominance to a source within a fragment, allowing for situations where sources might be equally energetic. This process aims to create fragments that are completely dominated by a single source.
Datasets
- Training: Pitch dynamics models (for pitch doubling/halving and voicing) were estimated from the AURORA 2 training set [4].
- Evaluation: Experiments were conducted using simultaneous speaker data constructed from the Grid corpus [7]. This corpus consists of read sentences, and utterances were artificially mixed at various Target-to-Masker Ratios (TMRs). The evaluation involved pairs of utterances that were same speaker, same gender (different speakers), and mixed gender.
Key Insights and Ideas
- Coherence is King: The paper strongly emphasizes that the coherence of a speech fragment (how completely it belongs to a single source) is more critical for downstream tasks like ASR than simply the size or completeness of the pitch track.
- Pitch Doubling/Halving is a Major Hurdle: Explicitly modeling these common pitch errors significantly improves the ability to track continuous pitch segments, even when the fundamental frequency is not clearly present.
- Soft Decisions for Robustness: Using soft assignments for source dominance in fragments makes the system more robust to ambiguity and overlapping speech.
- Bottom-up Segmentation Informs Top-down Processing: The MPT provides a robust bottom-up segmentation that can then be used by higher-level systems (like the Speech Fragment Decoder mentioned) for more accurate recognition.
Key Results
- Improved Coherence: The proposed MPT, when compared to simpler grouping methods (no post-processing or basic clustering), produced fragments with higher coherence across different TMRs. For example, at 0 dB TMR, the MPT (referred to as NP in Table 1) achieved a coherence of 77.6%, significantly higher than other methods.
- Recognition Accuracy: Speech recognition experiments demonstrated a direct correlation between fragment coherence and accuracy. Completely coherent fragments (CC) consistently yielded higher recognition scores than fragments derived from complete tracks (CT) or voiced segments (VS), especially as TMR decreased. For instance, at -3 dB TMR, CC fragments achieved significantly better accuracy than VS and CT fragments.
- Robustness to Noise: The system maintained reasonable accuracy and coherence even at low TMRs (down to 0 dB), indicating its robustness in noisy, mixed-speech environments.
Limitations and Future Work
- Parameter Tuning: The authors acknowledge that the tuning parameters of the MPT, specifically the voicing threshold (Os) and the pitch doubling/halving threshold (dp₁), require careful selection. While dp₁ showed some flexibility, Os impacts the trade-off between segment smoothness and information retrieval.
- Markov Model Order: The current system uses a first-order Markov process for pitch dynamics. The authors suggest that extending this to a second-order Markov model could potentially improve pitch trajectory modeling.
- Distractor Modeling: The assumption that distractor points are independent and identically distributed (iid) noise is a simplification. Future work could explore modeling relationships between distractors and actual pitch, potentially through re-estimation of model parameters.
- Generalization: While the system is designed to handle an arbitrary number of sources, the current study focuses on two-speaker mixtures. Further evaluation with more complex scenarios would be beneficial.
Summary updated