Introduction
In human speech, the placement of pauses is not arbitrary; it is fundamentally governed by the underlying syntactic structure of the sentence. For Text-to-Speech (TTS) systems to achieve natural-sounding prosody, they must accurately predict these pauses. However, a significant data bottleneck exists: while large-scale corpora with syntactic annotations (like the Penn Treebank) are abundant, speech datasets that include both syntactic trees and precise pause locations are rare and difficult to produce. This paper, originally presented at Interspeech 2015, proposes a joint model that leverages high-resource syntactic data to improve pause prediction in low-resource speech contexts.
Methodology: Latent Shift-Reduce Parsing
The researchers developed a transition-based dependency parser utilizing a modified shift-reduce algorithm. In a standard parser, the system moves through a sentence deciding whether to "shift" a word onto a stack or "reduce" words on the stack to form grammatical links. The authors extended this framework by bifurcating the standard SHIFT action into two distinct operations: SNONE (shift with no preceding pause) and SPAUSE (shift with a preceding pause).
To bridge the gap between text-only and speech-annotated data, the model employs a Structured Perceptron with Latent Variables. When training on standard text data (like the Wall Street Journal corpus) that lacks pause labels, the model treats the pause locations as "latent" or hidden variables. It essentially infers the most probable pause distribution that aligns with the known syntactic structure. This allows the model to learn the syntactic triggers for pauses even from data that was never intended for prosodic analysis.
Experimental Setup
The study utilized two primary datasets:
- Wall Street Journal (WSJ) Section of the Penn Treebank: Approximately 40,000 sentences providing dense syntactic information without pause labels.
- NAIST-NTT TED Talk Treebank: A smaller corpus of roughly 800 training sentences containing both syntactic structures and manually verified audio pauses.
By training on the TED data alone and then incrementally adding WSJ data, the researchers could measure the impact of the latent variable approach on pause prediction accuracy.
Results and Discussion
The performance of the model was evaluated using the F-measure for pause prediction. The results demonstrated a dramatic improvement over traditional methods:
- Baseline (Decision Tree): 6.28% F-measure.
- Joint Model (TED data only): 17.41% F-measure.
- Joint Model (Full TED + WSJ data): 28.96% F-measure.
As shown in the figure above, the inclusion of syntax-only data led to an 11.6% absolute gain in pause prediction accuracy. Crucially, the researchers found that this joint task did not degrade the parser's primary function; the Unlabeled Attachment Score (UAS) remained stable at approximately 91% for the WSJ data and 85% for the TED data.
Conclusion
This work demonstrates that syntax is a powerful proxy for prosody. By treating pauses as latent variables, the model successfully "borrows" knowledge from high-resource syntactic tasks to solve low-resource speech problems. While an F-measure of 28.96% indicates that pause prediction remains a complex challenge, the success of this joint modeling approach opens doors for other prosodic features—such as pitch, duration, and disfluency detection—to be learned through similar latent variable frameworks.
Summary updated