TL;DR
This paper compares four speech segmentation methods, finding that unconstrained, model-based approaches outperform traditional constrained methods when using canonical phone transcripts. Specifically, Artificial Neural Network (ANN) based segmentation emerged as the top performer, even rivaling constrained methods that use exact phone transcripts.
What Problem Does This Paper Address?
Accurately identifying the boundaries of speech sounds (like phones) within an audio signal is crucial for many speech processing tasks, including speech recognition and synthesis. The challenge lies in the inherent variability of speech due to co-articulation, where sounds influence their neighbors, making precise boundary detection difficult. Traditional "constrained" methods rely on having an exact phone transcript of the speech, which is costly and time-consuming to obtain. While using a "canonical" (standard pronunciation) transcript is more feasible, it introduces errors (insertions, deletions, substitutions) that degrade performance. This paper investigates whether "unconstrained" methods, which don't require explicit phone transcripts, can achieve comparable or better results.
Methods
The authors compare four distinct approaches:
- HMM Forced-Alignment (HMM-FA): A constrained, model-based approach. It uses Hidden Markov Models (HMMs) and requires a canonical phone transcript to align with the speech signal.
- HMM Phone-Loop (HMM-PL): An unconstrained, model-based approach. It uses HMMs but assumes all phones are equally likely, thus not needing a transcript for segmentation. It does use canonical transcripts for training the HMM models themselves.
- Group-Delay Function (GDF)-based Segmentation: A heuristic, unconstrained approach. It focuses on acoustic cues (specifically, changes in energy across different frequency bands) to detect transient behaviors at phone boundaries, without relying on any phone transcript or model training.
- Artificial Neural Network (ANN)-based Segmentation: An unconstrained, model-based approach. It uses a multi-layer feed-forward neural network trained as a boundary/non-boundary classifier. The input to the ANN is an augmented feature vector (current frame plus neighboring frames), and the output is a "boundary evidence" score.
Datasets
- TIMIT Corpus: Used for both training and testing. This is a widely used corpus for speech research, recorded in a clean environment with 16 kHz sampling rate and manually labeled phone transcripts. The authors exclude "SA" files.
- TIMIT-test: The test portion of the TIMIT corpus.
- Indian English (INE) test-set: A newly created test set of 10 TIMIT test prompts recorded by two speakers with Indian English accents. This was used to evaluate performance on non-native English.
- Telugu (TEL) test-set: A newly created test set of 20 Telugu sentences spoken by native Telugu speakers. This was used to evaluate performance on a different language, noting that only HMM-PL, GDF, and ANN could be tested as they don't rely on English phone sets.
Key Insights and Ideas
- Unconstrained is not necessarily worse: The study demonstrates that unconstrained, model-based approaches (HMM-PL and ANN) can outperform constrained HMM-FA when using canonical transcripts, suggesting that the errors introduced by canonical transcripts are a significant bottleneck.
- ANN shines in unconstrained segmentation: Among the unconstrained methods, the ANN-based approach consistently performed best, highlighting its ability to learn complex acoustic patterns for boundary detection without explicit transcript guidance.
- Canonical transcript errors matter: The paper explicitly shows that canonical transcripts have a significant error rate (27.7% Phone Error Rate on TIMIT training data) and that these errors directly impact the performance of HMM-FA.
- ANN generalizes well: The ANN-based models, trained on TIMIT, showed promising results on both Indian English and Telugu test sets, indicating good generalization capabilities.
Key Results
- Overall Performance: ANN-based segmentation achieved the best performance across most metrics (lower RMS Error, lower Boundary Error Rate, higher Agreement Percentage) on the TIMIT-test set, with an RMS Error of 9.3 ms and 92.18% agreement within 20 ms.
- Comparison to Constrained: ANN's performance was comparable to constrained methods using exact phone transcripts, especially for tolerances greater than 10 ms.
- HMM-FA vs. HMM-PL: HMM-PL (unconstrained) generally outperformed HMM-FA (constrained) when using canonical transcripts, especially in terms of Boundary Error Rate (BER).
- GDF Performance: The GDF-based heuristic approach showed competitive results, particularly on the Telugu test set, but was generally outperformed by ANN.
- Error Analysis: The paper provides detailed analysis of boundary deletion and insertion percentages for different phonetic classes, showing that ANN had the fewest insertions across classes.
Limitations and Future Work
- Canonical Transcript Errors: The paper acknowledges that the errors in canonical phone transcripts are a significant limitation for HMM-FA.
- ANN Generalization: While ANN showed good generalization, the paper notes that its performance on exact phone transcript comparisons was only comparable for larger tolerances (e.g., >10ms). Further work could explore improving this.
- Future Directions: The authors suggest that ANN-based models trained on TIMIT can be used for segmenting non-native English and other languages, implying potential for cross-lingual applications. The detailed error analysis also opens avenues for targeted improvements in specific phonetic class boundary detection.
Summary updated