Automatic measurement of vowel duration via structured prediction

Yossi Adi, Joseph Keshet, Emily Cibelli, Erin Gustafson, Cynthia G. Clopper, Matthew Goldrick

TL;DR

Measuring vowel duration by hand is a massive bottleneck for phonetic research, while existing automated tools require exact text transcripts to work. This paper introduces a machine learning model based on structured prediction that automatically extracts vowel durations directly from raw audio. It outperforms the current gold standard (HMM-based forced alignment) and is accurate enough to successfully replicate the scientific conclusions of statistical models built on manual human annotations.

What Problem Does This Paper Address?

In acoustic phonetics, understanding how and why vowel duration changes is a foundational area of study. Historically, researchers have had to manually annotate vowel boundaries (onsets and offsets) by staring at spectrograms. This is incredibly labor-intensive, limits the size of datasets, and introduces subjective human bias.

The current automated alternative is "forced alignment," which uses Hidden Markov Models (HMMs) to align a known text transcript with the audio. However, forced alignment has a major catch: it strictly requires an orthographic or phonetic transcript of the speech, and it often struggles without heavy data preprocessing. The authors set out to build a tool that can automatically and reliably measure vowel duration from an audio snippet without needing a transcript, making large-scale, replicable phonetic studies much easier.

Methods

The authors frame vowel measurement as a structured prediction problem. Instead of classifying each audio frame individually, the model takes an arbitrary-length acoustic signal (specifically, a consonant-vowel-consonant, or CVC, segment) and predicts a structured output: an onset-offset time pair.

Key aspects of their approach include:

  • Acoustic Features: They map the audio to a vector space using standard acoustic features (energy, pitch, zero-crossings, spectral changes).
  • Phoneme Classifier Feature: In a clever twist, they also feed the model the output of a pre-trained phoneme classifier. The probability that a given frame is a vowel or a nasal acts as a powerful guiding feature for the main algorithm.
  • Direct Loss Minimization (DLM): The model is trained using a DLM algorithm. Instead of optimizing a proxy metric, it directly minimizes the expected discrepancy (in milliseconds/frames) between the predicted vowel boundaries and the manual human annotations.

Datasets

The researchers evaluated their model using three distinct phonetic datasets:

  1. Heller and Goldrick, 2014 (HG): 2,395 CVC noun audio segments from 64 speakers.
  2. Clopper and Tamati, 2014 (CT): 777 CVC tokens from 20 speakers, featuring different American English dialects.
  3. Hillenbrand et al., 1995 (HGCW): A classic dataset of 12 different vowels from 139 speakers (men, women, and children). This was used strictly as a hold-out test set to evaluate how the models perform on completely unseen data.

Key Insights and Ideas

  • Transcript Independence is a Game Changer: Because the DLM model doesn't need a text transcript to find the vowel, it opens the door to analyzing naturalistic, un-transcribed speech, or dialects that don't map cleanly to standard dictionary pronunciations.
  • Direct Optimization Beats Generative Alignment: By directly penalizing the model for boundary errors (via DLM) rather than relying on the generative path-finding of an HMM, the algorithm achieves much tighter alignment with human annotators.
  • The "Scientific Replication" Test: The authors didn't just measure raw accuracy; they ran mixed-effects regression models on their automated outputs to see if they would yield the same scientific conclusions as the manual data. This is a brilliant, practical benchmark for any tool aimed at researchers.

Key Results

  • Lower Measurement Deviation: When trained and tested on the same corpus, the DLM model crushed the HMM baseline. For the HG dataset, the DLM's average onset deviation was 5.21 ms, compared to the HMM's 16.67 ms.
  • Robust to Mismatched Data: Even when the DLM was trained on one dataset and tested on another (e.g., trained on CT, tested on HG), it still outperformed the HMM. DLM onset deviation was 9.84 ms vs. the HMM's 35.90 ms.
  • Successful Statistical Replication: When the authors ran inferential statistical models on the HG dataset, the DLM-generated measurements successfully recovered the same significant fixed effects (e.g., the effect of production context on vowel duration) as the manual human measurements.
  • Classifier Impact: While the pre-trained phoneme classifier boosted performance, the DLM model without the classifier still generally outperformed the HMM baseline.

Limitations and Future Work

The primary limitation of the current system is its scope: it is designed specifically for single vowels in a CVC (consonant-vowel-consonant) context, which is common in laboratory stimuli but not reflective of continuous, real-world speech.

For future work, the authors plan to extend this approach to multisyllabic words and more naturalistic speech. To achieve this without relying on transcripts, they suggest combining their structured prediction scheme with modern sequence-based deep learning architectures, like Recurrent Neural Networks (RNNs).

Summary updated

Abstract

A key barrier to making phonetic studies scalable and replicable is the need to rely on subjective, manual annotation. To help meet this challenge, a machine learning algorithm was developed for automatic measurement of a widely used phonetic measure: vowel duration. Manually-annotated data were used to train a model that takes as input an arbitrary length segment of the acoustic signal containing a single vowel that is preceded and followed by consonants and outputs the duration of the vowel. The model is based on the structured prediction framework. The input signal and a hypothesized set of a vowel's onset and offset are mapped to an abstract vector space by a set of acoustic feature functions. The learning algorithm is trained in this space to minimize the difference in expectations between predicted and manually-measured vowel durations. The trained model can then automatically estimate vowel durations without phonetic or orthographic transcription. Results comparing the model to three sets of manually annotated data suggest it outperformed the current gold standard for duration measurement, an hidden Markov model-based forced aligner (which requires orthographic or phonetic transcription as an input).
BibTeX
@article{adi2016automatic,
  author = {Yossi Adi and Joseph Keshet and Emily Cibelli and Erin Gustafson and Cynthia G. Clopper and Matthew Goldrick},
  title = {{Automatic measurement of vowel duration via structured prediction}},
  journal = {The Journal of the Acoustical Society of America},
  year = {2016},
  doi = {10.1121/1.4972527},
}

Loading related content...

Figure