Abstract
A key barrier to making phonetic studies scalable and replicable is the need to rely on subjective, manual annotation. To help meet this challenge, a machine learning algorithm was developed for automatic measurement of a widely used phonetic measure: vowel duration. Manually-annotated data were used to train a model that takes as input an arbitrary length segment of the acoustic signal containing a single vowel that is preceded and followed by consonants and outputs the duration of the vowel. The model is based on the structured prediction framework. The input signal and a hypothesized set of a vowel's onset and offset are mapped to an abstract vector space by a set of acoustic feature functions. The learning algorithm is trained in this space to minimize the difference in expectations between predicted and manually-measured vowel durations. The trained model can then automatically estimate vowel durations without phonetic or orthographic transcription. Results comparing the model to three sets of manually annotated data suggest it out-performed the current gold standard for duration measurement, an HMM-based forced aligner (which requires orthographic or phonetic transcription as an input).
Introduction
In acoustic phonetics, understanding how and why vowel duration changes is a foundational area of study. Historically, researchers have had to manually annotate vowel boundaries (onsets and offsets) by examining spectrograms. This process is labor-intensive, limits dataset size, and introduces subjective human bias.
The current automated alternative, "forced alignment," utilizes Hidden Markov Models (HMMs) to align known text transcripts with audio. However, forced alignment strictly requires orthographic or phonetic transcripts and often necessitates heavy data preprocessing. This paper introduces a machine learning model based on structured prediction that automatically extracts vowel durations directly from raw audio without needing a transcript, facilitating large-scale, replicable phonetic studies.
Methods: Structured Prediction for Vowel Measurement
The authors frame vowel measurement as a structured prediction problem. Instead of classifying each audio frame individually, the model takes an arbitrary-length acoustic signal—specifically, a consonant-vowel-consonant (CVC) segment—and predicts a structured output: an onset-offset time pair ($t_b, t_e$).
Key aspects of the approach include:
- Acoustic Features: The audio is mapped to a vector space using standard features such as energy, pitch, zero-crossings, and spectral changes.
- Phoneme Classifier Feature: The model incorporates the output of a pre-trained phoneme classifier. The probability that a given frame is a vowel or a nasal acts as a powerful guiding feature.
- Direct Loss Minimization (DLM): The model is trained using a DLM algorithm to directly minimize the expected discrepancy (in milliseconds or frames) between predicted vowel boundaries and manual human annotations, rather than optimizing a proxy metric.
Experimental Setup and Datasets
The model was evaluated using three distinct phonetic datasets:
- Heller and Goldrick (2014): 2,395 CVC noun segments from 64 speakers.
- Clopper and Tamati (2014): 777 CVC tokens from 20 speakers across different American English dialects.
- Hillenbrand et al. (1995): A classic dataset of 12 vowels from 139 speakers, used as a hold-out test set to evaluate generalization to unseen data.
Key Results
The DLM model demonstrated significant improvements over the HMM-based forced alignment gold standard:
- Lower Measurement Deviation: On the HG dataset, the DLM's average onset deviation was 5.21 ms, compared to the HMM's 16.67 ms. Even in mismatched data scenarios (e.g., training on CT and testing on HG), the DLM outperformed the HMM (9.84 ms vs. 35.90 ms deviation).
- Scientific Replication: Mixed-effects regression models run on the DLM-generated measurements successfully recovered the same significant fixed effects (such as the effect of production context) as those derived from manual human measurements.
- Transcript Independence: By eliminating the need for text transcripts, the model enables the analysis of naturalistic, un-transcribed speech or dialects that deviate from standard dictionary pronunciations.
Conclusion
The proposed structured prediction model offers a robust, transcript-independent solution for measuring vowel duration in CVC contexts. While currently limited to single-vowel segments, future work aims to extend this approach to multisyllabic words and continuous speech by integrating the structured prediction scheme with sequence-based deep learning architectures like Recurrent Neural Networks (RNNs).
Summary updated