Signal-based accent and phrase marking using the fujisaki model

Hussein Hussein, Rüdiger HoffmannInterspeech 2010

TL;DR

This paper presents an automatic algorithm for marking accents and phrases in speech using the Fujisaki model, a quantitative approach to intonation. By analyzing the fundamental frequency (F0) contour, the algorithm aims to reduce the time-consuming manual process of prosodic annotation, achieving promising success rates that, while not yet state-of-the-art, offer a solid foundation for further development.

What Problem Does This Paper Address?

Accurate prosodic marking – identifying where accents (emphasis) and phrase boundaries occur in speech – is crucial for many speech processing tasks, including synthesis and recognition. Traditionally, this marking is done manually by human annotators, a process that is not only time-consuming but also subjective and prone to inconsistencies. This paper tackles the challenge of automating this process, aiming to create a more objective and efficient method for prosodic annotation by leveraging acoustic features.

Methods

The core of the proposed method lies in analyzing the F0 contour (the fundamental frequency, closely related to pitch) of speech signals. The authors employ the Fujisaki model, a well-established quantitative model of intonation, to break down the F0 contour into its constituent accent and phrase commands.

Key steps include:

  • Pitch Mark Extraction: A hybrid algorithm combines outputs from multiple pitch marking algorithms to accurately identify the F0 contour.
  • F0 Contour Calculation: The F0 contour is calculated based on the distance between pitch marks.
  • Fujisaki Parameter Extraction: Two distinct algorithms (Mixdorff and Kruschke) are used to extract the Fujisaki parameters from the smoothed F0 contour. These parameters represent the underlying prosodic structure. The Kruschke algorithm, which uses piecewise polynomial approximation and Continuous Wavelet Transform (CWT), is found to perform better.
  • Accent and Phrase Annotation: Based on the extracted Fujisaki parameters, the algorithm automatically marks accents (major/minor) and phrases (major/minor) at the word level. Specific rules are applied to combine accent commands into word-level accents and to define phrase boundaries based on phrase commands, pauses, and word durations.

Datasets

The primary dataset used is a subset of the multilingual Verbmobil database. This subset consists of 10 hours of spontaneous German speech from multiple male and female speakers. For evaluation, a smaller subset of the Verbmobil database was manually annotated by a non-native German speaker to serve as a reference for accents and phrases.

Key Insights and Ideas

  • Fujisaki Model as a Foundation: The paper demonstrates the viability of using the Fujisaki model's quantitative parameters as a basis for automatic prosodic marking, moving beyond purely rule-based or statistical methods.
  • Combining Accent/Phrase Levels: Combining minor and major accents into a single level, and similarly for phrases, significantly improves the success rate of the automatic marking algorithm. This suggests that a more generalized prosodic marking might be more robust for automated systems.
  • Algorithm Optimization: The authors emphasize that optimizing the algorithm's parameters is crucial for achieving the best results, highlighting the delicate balance between success rate and accuracy.

Key Results

The proposed algorithm achieved the following results on the Verbmobil subset:

  • Accent Marking: A success rate (SR) of 77.11% and an accuracy (ACC) of 27.46% when combining minor and major accents.
  • Phrase Marking: A success rate (SR) of 67.12% and an accuracy (ACC) of 22.97% when combining minor and major phrases.

These results are promising but are noted as not yet reaching the state-of-the-art, which reports higher SR for accents (82.6%) and phrases (88.3%).

Limitations and Future Work

  • Performance Gap: The authors acknowledge that their results are not yet as good as the state-of-the-art, indicating room for improvement in accuracy.
  • Manual Correction Still Needed: The paper concludes that manual correction of the automatic labeling output is still necessary to achieve high-quality speech databases.
  • Future Directions: While not explicitly detailed, the paper implies that further refinement of the Fujisaki parameter extraction and the annotation rules could lead to improved performance. Exploring different Fujisaki model implementations or integrating other acoustic features might also be beneficial.

Summary updated

BibTeX
@inproceedings{hussein2010signal,
  author = {Hussein Hussein and Rüdiger Hoffmann},
  title = {{Signal-based accent and phrase marking using the fujisaki model}},
  booktitle = {Interspeech},
  year = {2010},
}
Figure