On the automatic toBI accent type identification from data

César González-Ferreras, Carlos Vivaracho-Pascual, David Escudero-Mancebo, Valentín Cardeñoso-PayoInterspeech 2010

TL;DR

This paper tackles the challenging task of automatically identifying specific ToBI accent types in speech, moving beyond simpler "accented vs. unaccented" distinctions. By analyzing the Boston Radio News Corpus and employing both decision trees and neural networks, they demonstrate that while multiclass accent identification is difficult, incorporating expressive prosodic features like Bézier curve coefficients significantly improves accuracy.

What Problem Does This Paper Address?

Accurately identifying prosodic events in speech is crucial for understanding meaning, disambiguating utterances, and improving speech technologies like text-to-speech systems. The ToBI (ToBI) labeling system is a widely used framework for annotating prosody. While previous work has focused on distinguishing between "accented" and "unaccented" speech, this paper addresses the more ambitious and challenging problem of automatically identifying specific ToBI accent types. This multiclass classification is significantly harder due to imbalanced class distributions (some accent types are much rarer than others) and inherent similarities between certain accent types, making them difficult for machines (and sometimes humans) to distinguish.

Methods

The authors explore two main machine learning approaches:

  • C4.5 Decision Trees (DT): A classic tree-based classifier.
  • Multilayer Perceptron (MLP) Neural Networks (NN): A more complex, connectionist model.

Both classifiers were trained using stratified 10-fold cross-validation. To address the issue of imbalanced data, they employed a data-level technique: minority class example repetition (oversampling).

A key methodological contribution is the investigation of novel prosodic features. Beyond standard acoustic features (F0, energy, duration) and pseudolinguistic features (Part-of-Speech tags), they incorporated Bézier function fitting coefficients. These coefficients capture the temporal evolution of the F0 contour within intonation units (syllables or words), providing a more expressive representation of prosodic patterns. They also explored different methods for normalizing features and encoding categorical POS tags.

Datasets

The primary dataset used is the Boston University Radio News Corpus. This corpus contains phonetic, syllabic, and word-level annotations, along with ToBI accent labels and their positions. The authors specifically focused on the 7 most frequent accent tones (H, L+H, !H, H+!H, L+!H, L, L*+H), excluding less frequent or undetermined marks. They acknowledge that this corpus, like most ToBI-tagged corpora, suffers from imbalanced class distributions.

Key Insights and Ideas

  • Multiclass ToBI identification is significantly harder than binary accent detection: Accuracy drops considerably when trying to distinguish between specific accent types.
  • Imbalanced data is a major hurdle: Classifiers tend to become specialized in recognizing the more frequent accent types, leading to poor performance on rarer ones.
  • Inherent class similarity contributes to confusion: Some ToBI accent types are conceptually similar, making them difficult to differentiate even for human annotators, which translates to machine learning challenges.
  • Expressive prosodic features are crucial: Incorporating features that capture the temporal dynamics of the F0 contour, such as Bézier coefficients, significantly boosts recognition accuracy, especially for the multiclass problem.
  • MLPs generally outperform Decision Trees: For this task, neural networks showed better robustness, particularly in handling the imbalanced data and achieving better performance across a wider range of classes.
  • Pairwise analysis reveals specific confusions: Analyzing the confusion matrix for pairs of accent types helps pinpoint which accents are most frequently mistaken for each other, providing insights for future model improvements.

Key Results

  • Accent vs. No Accent: Achieved accuracies of 84.7% with NN and 82.7% with DT, aligning with state-of-the-art results.
  • Multiclass Accent Type Identification: Accuracy dropped significantly. For example, the !H* accent saw its rate increase from 18.7% to 29.3% with the use of Bézier coefficients.
  • Impact of Bézier Coefficients: The use of Bézier coefficients consistently improved performance in both multiclass and pairwise classification tasks, though the improvement was marginal for the binary accent vs. no accent task.
  • Class-specific performance: While the "none" (unaccented) class was discriminated with high accuracy, confusion was high among other accent types. The DT classifier showed sensitivity to class imbalance, while the NN compensated better for the rest of the classes.

Limitations and Future Work

  • Imbalanced Data: The paper acknowledges the persistent challenge of imbalanced classes in ToBI corpora and its impact on classifier performance.
  • Class Similarity: The inherent conceptual similarity between certain ToBI accents remains a significant obstacle.
  • Future Work:
    • Investigate hierarchical recognition schemes where different classifiers specialize in separating specific clusters of ToBI accents.
  • Explore alternative prosodic feature parametrization techniques (e.g., Fujisaki, Tilt, IntSint) to further improve discrimination.
  • Consider expert fusion to combine results from different classifiers or units (words vs. syllables) to leverage their respective strengths.
  • The authors suggest that combining a-priori judgments with a-posteriori in-sequence evidence might help separate acoustically similar but conceptually distinct accents.

Summary updated

BibTeX
@inproceedings{gonzalezferreras2010automatic,
  author = {César González-Ferreras and Carlos Vivaracho-Pascual and David Escudero-Mancebo and Valentín Cardeñoso-Payo},
  title = {{On the automatic toBI accent type identification from data}},
  booktitle = {Interspeech},
  year = {2010},
}
Figure