An Automatic Prosody Tagger for Spontaneous Speech

Mónica Domínguez, Mireia Farrús, Leo WannerCOLING 2016

TL;DR

This paper introduces a novel, language-independent annotation schema and an automatic prosody tagger designed to analyze spontaneous speech. By focusing on acoustic parameters at the prosodic phrase level, they achieve satisfactory inter-annotator agreement and a tagger that outperforms a baseline in detecting prosodic boundaries and prominence, paving the way for more natural human-machine interaction.

What Problem Does This Paper Address?

Understanding and generating natural-sounding speech is crucial for advanced communication technologies. While speech prosody (intonation, rhythm, stress) plays a vital role in conveying meaning, emotion, and structure, empirical research has been hampered by a lack of large-scale, well-annotated prosody resources. Existing annotation conventions, like ToBI, are often language-dependent and subjective, leading to low inter-annotator agreement and making it difficult to train and evaluate computational models. Furthermore, current methods often rely on textual or linguistic units, which are insufficient for capturing the complexities of spontaneous speech, such as hesitations and false starts. This paper aims to bridge this gap by developing a more objective, acoustic-based annotation schema and an automatic tagger for spontaneous speech.

Methods

The authors propose a two-pronged approach:

  1. A Parametric Annotation Schema: This schema is designed to be language-independent and focuses on acoustic features at the prosodic phrase (PPh) level. It leverages normalized acoustic parameters (F0, intensity, duration) to define prosodic boundaries and prominence. The schema is implemented as an extension of the Praat software, allowing for feature annotation and retrieval within the same environment.
  2. An Automatic Prosody Tagger: This tagger is built upon the proposed schema and operates in a modular fashion. It processes speech signals to extract acoustic features at different levels of abstraction (utterance, voiced segments, PPhs). The tagger then uses these features to predict PPh boundaries and identify prominent words within those phrases. A key aspect is its ability to handle spontaneous speech by incorporating disfluencies into the closest PPh and using acoustic cues rather than solely relying on textual information. The tagger can operate with or without pre-existing word segmentation.

Datasets

The evaluation of the proposed methodology and tagger was conducted on five spontaneous speech samples:

  • Three dialogues in Spanish (medical context).
  • Two monologues in English (biographical introductions).

These samples were manually annotated by two expert annotators following the developed guidelines. The paper does not introduce a new benchmark dataset but rather uses existing speech samples for evaluation.

Key Insights and Ideas

  • Acoustic-based prosody is key: The paper strongly advocates for using acoustic parameters (F0, intensity, duration) as the primary basis for prosody annotation, moving away from purely linguistic or subjective labels.
  • Prosodic Phrase (PPh) as the unit of analysis: Focusing on the PPh level proves effective for capturing hierarchical prosodic structure and correlates with thematic structure, making it a valuable unit for computational analysis.
  • Normalization is crucial: Normalizing acoustic features (e.g., using z-scores) across different levels of abstraction (utterance, voiced segment) helps to create a more robust and comparable representation of prosodic cues.
  • Handling spontaneous speech complexities: The methodology explicitly addresses challenges like disfluencies and pauses, integrating them into the annotation process rather than treating them as noise, which is vital for real-world applications.
  • Modular architecture for flexibility: The tagger's modular design allows for easier optimization, extension (e.g., to prosodic word detection), and integration of new features or models.

Key Results

  • Inter-annotator Agreement: The annotation schema achieved satisfactory Cohen's kappa values, ranging from 0.51 to 0.98, with higher agreement for boundary labeling than prominence labeling.
  • Tagger Performance: The automatic prosody tagger demonstrated accuracy rates above 0.84 for both prominence and boundary detection.
  • Recall Improvement: The tagger significantly outperformed a baseline (using only F0 cues) in recall for both tasks, indicating its ability to identify more true positives.
  • F-measure: The tagger achieved better overall F-measure scores compared to the baseline, suggesting a good balance between precision and recall.
  • Language Independence: The results suggest the methodology is flexible enough to support language-independent speech signal analysis.

Limitations and Future Work

  • Corpus Size: The authors acknowledge that the corpus used for evaluation is relatively small, which might limit definitive conclusions about language dependency of prominence annotation.
  • Neutral Speech: The methodology, based on deviations of normalized values, might face challenges in accurately tagging neutral speech where prosodic variations are minimal.
  • Future Work: The authors plan to extend the implementation for detecting smaller prosodic units (like prosodic words) and explore the integration of linguistic features (e.g., part-of-speech tags, syntactic dependencies) with acoustic features. They also propose re-implementing the tagger as a neural network application for potentially enhanced prediction.

Summary updated

BibTeX
@inproceedings{dominguez2016automatic,
  author = {Mónica Domínguez and Mireia Farrús and Leo Wanner},
  title = {{An Automatic Prosody Tagger for Spontaneous Speech}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2016},
}
Figure