TL;DR
This paper introduces an open-source tool and dataset for crowdsourcing speech prominence (word emphasis) annotations, alongside a neural network estimator that generalizes remarkably well to unseen speakers and conversational styles. Crucially, the authors provide data-collection guidelines showing that for a fixed budget, training on more unique utterances with fewer annotations per word yields better model performance than having high annotator consensus on a smaller dataset.
What Problem Does This Paper Address?
When we speak, certain words naturally stand out due to changes in pitch, loudness, or rhythm—a phenomenon known as speech prominence. Accurately identifying these emphasized words is essential for downstream tasks like emotion recognition, text summarization, and building expressive text-to-speech (TTS) systems.
However, determining which words are prominent is highly subjective and traditionally requires expensive, time-consuming manual annotation by linguistic experts. While automated, rule-based acoustic heuristics exist, they struggle to capture the high-dimensional complexity of human perception and fail to generalize to new datasets. Machine learning models offer a solution but are bottlenecked by the lack of large, diverse, open-source human-annotated datasets.
Methods
The authors tackle this problem with a two-pronged approach: scalable crowdsourcing and optimized neural architecture design.
- Crowdsourced Annotation: They developed a word-selection interface built on top of ReSEval (Reproducible Subjective Evaluation). Non-expert native speakers listen to audio clips and click on words they perceive as emphasized. To combat the rampant issue of survey bots, the authors implemented strict filtering rules (e.g., excluding annotators who flag more than two-thirds of words as emphasized across multiple utterances).
- Neural Prominence Estimator: The proposed model takes an 80-channel Mel spectrogram as input and processes it through a 6-layer convolutional network. Because audio features are frame-based (e.g., every 10 milliseconds) and prominence labels are word-based, the network must downsample features from frame resolution to word resolution.
The authors experimented with four downsampling locations and four mathematical methods (Average, Center, Max, Sum) to find the optimal configuration.
Datasets
The study utilizes two main datasets:
- LibriTTS (train-clean-100 partition): The authors collected crowdsourced annotations for one-eighth of this dataset, totaling 3,626 utterances (6.42 hours of speech across 18 speakers). This newly labeled dataset is open-sourced under a CC-BY-4.0 license.
- Buckeye Corpus: A dataset of conversational American English. The authors used a subset of 256 utterances annotated by 32 native speakers strictly for zero-shot evaluation to test how well their model generalizes to completely unseen speakers and highly casual speaking styles.
Key Insights and Ideas
- Context Matters for Downsampling: Performing variable-stride downsampling within the neural network (Intermediate wordwise) significantly outperforms segmenting the audio into isolated words before feeding it to the network (Prehoc). This indicates that the convolutional layers benefit from seeing acoustic context across word boundaries.
- Data Quantity Trumps Label Redundancy: If you have a fixed budget for human annotation, the optimal strategy is to collect one annotation for as many unique utterances as possible, rather than multiple overlapping annotations for a smaller set of utterances. The diversity of data helps the neural network learn generalizable patterns, which outweighs the variance reduction provided by multiple annotators.
- Redundancy Accelerates Convergence: While having multiple annotators per utterance doesn't improve final model accuracy for a fixed budget, it drastically reduces the number of training steps required for the model to converge.
Key Results
- State-of-the-Art Generalization: When evaluated on the unseen, conversational Buckeye dataset, the proposed model achieved a Pearson correlation of 0.675, vastly outperforming a top-performing wavelet-based heuristic baseline (0.529).
- Efficiency Gains: The optimal architecture configuration—using Intermediate wordwise downsampling with the Sum method—proved to be 39.7% faster to train and reduced GPU memory consumption by 69.1% compared to the previously standard prehoc segmentation method.
Limitations and Future Work
A primary limitation highlighted by the authors is the vulnerability of crowdsourcing platforms like Amazon Mechanical Turk to automated bots, which required aggressive filtering and cost them roughly 26% of their initial annotator pool. Additionally, the pipeline relies on an external forced aligner (P2FA) to establish word boundaries, meaning errors in alignment could propagate to the prominence estimator.
For future work, the authors suggest applying this crowdsourcing and modeling framework to other subjective, word-level speech attributes, such as disfluency, falsetto, and vocal fry.
Summary updated