Abstract
Training categorical prosody models for spoken language systems requires a significant amount of speech data annotated with the discrete labels of interest (such as boundary marks or word prominence information). In practice, the difficulty and expense incurred in producing corpora with rich prosodic transcriptions severely limits their integration within applications. In this paper, we explore the possibility of using a large, unlabeled corpus to adapt, in an unsupervised fashion, acoustic-prosodic models trained from a small, human-annotated seed dataset. Our experiments show that the proposed adaptation scheme improves the ability of the acoustic-prosodic model to distinguish between prosodic categories. On a test set derived from the Boston University Radio News Corpus, the adapted models reduced pitch accent detection error rate by 4.3% relative to the seed acoustic-prosodic models trained from the annotated data.
Introduction
In speech technology, "prosody" refers to the rhythm, stress, and intonation of speech. Categorical prosody models, often based on the ToBI (Tones and Break Indices) standard, are vital for tasks like speech recognition and synthesis. However, a major bottleneck exists: training these models requires speech data meticulously annotated by human experts. Manual annotation is slow, expensive, and difficult to scale, often resulting in small datasets that lead to "data sparsity" and poor generalization. This research investigates whether vast amounts of unannotated speech data can be leveraged to improve a model trained on a small, labeled "seed" set.
Methodology
The authors propose an unsupervised adaptation scheme based on a modified version of the Expectation-Maximization (EM) and Maximum A-Posteriori (MAP) algorithms. The process begins with a baseline Gaussian Mixture Model (GMM) trained on a small human-labeled dataset.
Instead of generating "hard" labels for the unlabeled data, the seed model assigns a confidence weight to each sample in a large unlabeled corpus. In the developed weighted EM-MAP framework, samples with high confidence have a stronger influence on parameter updates, while low-confidence samples are marginalized. This approach allowed the researchers to transition from simple "diagonal covariance" GMMs to more complex "full covariance" GMMs, which the original small dataset could not support without overfitting.
The model utilizes a variety of acoustic features related to duration, fundamental frequency (F0), and energy (RMSE) to characterize prosodic events.
Experimental Setup
Two primary datasets were used:
- Labeled (Seed/Evaluation): The Boston University Radio News Corpus (BU-RNC), containing approximately 3 hours of broadcast news with ToBI-style annotations. A small subset was used for training to simulate data-starved conditions.
- Unlabeled (Adaptation): The WSJ1 (CSR II) corpus, providing roughly 52 hours of broadcast news speech (407,000 words) with audio and transcripts but no prosodic labels.
Key Results
The system was evaluated on a binary pitch accent detection task (identifying the presence or absence of an accent on a word). The results demonstrate that unsupervised adaptation significantly outperforms simple scaling of the seed model.
As shown in the table above, the adapted model (45-mixture, full covariance) achieved a 25.26% error rate on the test set, representing a 4.3% relative reduction in error compared to the best seed model (26.39%). This gain is significant as it was achieved without any additional human labeling effort.
Discussion and Future Work
The core insight of this research is that perfect labels are not required for model improvement; treating seed model outputs as probabilities allows for effective filtering of noise during unsupervised learning. Furthermore, unsupervised adaptation acts as a bridge, providing the data density necessary to train sophisticated model architectures that were previously unreachable due to data sparsity.
Future work may explore discriminative adaptation approaches, which focus on the boundaries between prosodic categories rather than just the data distribution. Additionally, testing the robustness of this weighting scheme across different domains (e.g., adapting news models using casual conversation) remains an important next step for integrating these models into full Automatic Speech Recognition (ASR) pipelines.
Summary updated