Crowdsourced and Automatic Speech Prominence Estimation

Max Morrison, Pranav Pawar, Nathan Pruyne, Jennifer Cole, Bryan PardoarXiv 2023 · arXiv

TL;DR

This paper introduces an open-source tool and dataset for crowdsourcing speech prominence (word emphasis) annotations, alongside a neural network estimator that generalizes remarkably well to unseen speakers and conversational styles. Crucially, the authors provide data-collection guidelines showing that for a fixed budget, training on more unique utterances with fewer annotations per word yields better model performance than having high annotator consensus on a smaller dataset.

What Problem Does This Paper Address?

When we speak, certain words naturally stand out due to changes in pitch, loudness, or rhythm—a phenomenon known as speech prominence. Accurately identifying these emphasized words is essential for downstream tasks like emotion recognition, text summarization, and building expressive text-to-speech (TTS) systems.

However, determining which words are prominent is highly subjective and traditionally requires expensive, time-consuming manual annotation by linguistic experts. While automated, rule-based acoustic heuristics exist, they struggle to capture the high-dimensional complexity of human perception and fail to generalize to new datasets. Machine learning models offer a solution but are bottlenecked by the lack of large, diverse, open-source human-annotated datasets.

Methods

The authors tackle this problem with a two-pronged approach: scalable crowdsourcing and optimized neural architecture design.

  1. Crowdsourced Annotation: They developed a word-selection interface built on top of ReSEval (Reproducible Subjective Evaluation). Non-expert native speakers listen to audio clips and click on words they perceive as emphasized. To combat the rampant issue of survey bots, the authors implemented strict filtering rules (e.g., excluding annotators who flag more than two-thirds of words as emphasized across multiple utterances).
Fig. 1. Crowdsourced human emphasis annotation interface | Annotators listen to a speech recording and click to boldface the words they perceive as emphasized.
  1. Neural Prominence Estimator: The proposed model takes an 80-channel Mel spectrogram as input and processes it through a 6-layer convolutional network. Because audio features are frame-based (e.g., every 10 milliseconds) and prominence labels are word-based, the network must downsample features from frame resolution to word resolution.

The authors experimented with four downsampling locations and four mathematical methods (Average, Center, Max, Sum) to find the optimal configuration.

Fig. 2. Three candidate prominence estimation models | We ex- periment with a framewise model (left) as well as two wordwise models: one that downsamples from frames to words just before the loss function (posthoc wordwise; center) and one that downsamples within the neural network (intermediate wordwise; right). The yel- low, framewise encoder is a stack of convolution layers that operate at the frame resolution. The magenta, wordwise decoder is a stack of convolution layers that operate at the word resolution.

Datasets

The study utilizes two main datasets:

  • LibriTTS (train-clean-100 partition): The authors collected crowdsourced annotations for one-eighth of this dataset, totaling 3,626 utterances (6.42 hours of speech across 18 speakers). This newly labeled dataset is open-sourced under a CC-BY-4.0 license.
  • Buckeye Corpus: A dataset of conversational American English. The authors used a subset of 256 utterances annotated by 32 native speakers strictly for zero-shot evaluation to test how well their model generalizes to completely unseen speakers and highly casual speaking styles.

Key Insights and Ideas

  • Context Matters for Downsampling: Performing variable-stride downsampling within the neural network (Intermediate wordwise) significantly outperforms segmenting the audio into isolated words before feeding it to the network (Prehoc). This indicates that the convolutional layers benefit from seeing acoustic context across word boundaries.
  • Data Quantity Trumps Label Redundancy: If you have a fixed budget for human annotation, the optimal strategy is to collect one annotation for as many unique utterances as possible, rather than multiple overlapping annotations for a smaller set of utterances. The diversity of data helps the neural network learn generalizable patterns, which outweighs the variance reduction provided by multiple annotators.
  • Redundancy Accelerates Convergence: While having multiple annotators per utterance doesn't improve final model accuracy for a fixed budget, it drastically reduces the number of training steps required for the model to converge.

Key Results

  • State-of-the-Art Generalization: When evaluated on the unseen, conversational Buckeye dataset, the proposed model achieved a Pearson correlation of 0.675, vastly outperforming a top-performing wavelet-based heuristic baseline (0.529).
  • Efficiency Gains: The optimal architecture configuration—using Intermediate wordwise downsampling with the Sum method—proved to be 39.7% faster to train and reduced GPU memory consumption by 69.1% compared to the previously standard prehoc segmentation method.
Ablations of downsampling methods and locations

Limitations and Future Work

A primary limitation highlighted by the authors is the vulnerability of crowdsourcing platforms like Amazon Mechanical Turk to automated bots, which required aggressive filtering and cost them roughly 26% of their initial annotator pool. Additionally, the pipeline relies on an external forced aligner (P2FA) to establish word boundaries, meaning errors in alignment could propagate to the prominence estimator.

For future work, the authors suggest applying this crowdsourcing and modeling framework to other subjective, word-level speech attributes, such as disfluency, falsetto, and vocal fry.

Summary updated

Abstract

The prominence of a spoken word is the degree to which an average native listener perceives the word as salient or emphasized relative to its context. Speech prominence estimation is the process of assigning a numeric value to the prominence of each word in an utterance. These prominence labels are useful for linguistic analysis, as well as training automated systems to perform emphasis-controlled text-to-speech or emotion recognition. Manually annotating prominence is time-consuming and expensive, which motivates the development of automated methods for speech prominence estimation. However, developing such an automated system using machine-learning methods requires human-annotated training data. Using our system for acquiring such human annotations, we collect and open-source crowdsourced annotations of a portion of the LibriTTS dataset. We use these annotations as ground truth to train a neural speech prominence estimator that generalizes to unseen speakers, datasets, and speaking styles. We investigate design decisions for neural prominence estimation as well as how neural prominence estimation improves as a function of two key factors of annotation cost: dataset size and the number of annotations per utterance.
BibTeX
@misc{morrison2023crowdsourced,
  author = {Max Morrison and Pranav Pawar and Nathan Pruyne and Jennifer Cole and Bryan Pardo},
  title = {{Crowdsourced and Automatic Speech Prominence Estimation}},
  year = {2023},
  eprint = {2310.08464},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2310.08464},
}

Loading related content...

Figure