Investigating syllabic prominence with conditional random fields and latent-dynamic conditional random fields

Francesco Cutugno, Enrico Leone, Bogdan Ludusan, Antonio OrigliaInterspeech 2012

TL;DR

This paper explores automatic detection of syllabic prominence, a key aspect of speech prosody. By comparing standard Conditional Random Fields (CRFs) with their extension, Latent-Dynamic CRFs (LDCRFs), the authors demonstrate that LDCRFs significantly improve prominence detection accuracy. They also investigate the impact of different feature sets and context window sizes, finding that syllable length and a novel pitch movement feature are particularly beneficial.

What Problem Does This Paper Address?

Syllabic prominence, the emphasis placed on certain syllables within a word or utterance, is a fundamental component of speech prosody. It plays a crucial role in conveying meaning, emotion, and linguistic structure. While linguists have long studied prominence, automatically identifying it in speech remains a challenge. Existing automatic systems often rely on rule-based approaches, which can be difficult to interpret and adapt. Machine learning methods, particularly sequence labeling models like CRFs, offer a data-driven alternative. However, previous work has primarily focused on pitch accent detection at the word level, often incorporating complex linguistic features. This paper aims to bridge this gap by systematically investigating syllabic prominence detection at the syllable level using acoustic features and comparing the effectiveness of standard CRFs against a more advanced model, LDCRFs, while also exploring the influence of feature engineering and context.

Methods

The core of this research lies in comparing two sequence labeling models:

  • Conditional Random Fields (CRFs): A standard discriminative model for sequence labeling tasks. CRFs model the conditional probability of a sequence of labels given a sequence of observations, considering both state-feature functions (relating observations to labels) and transition-feature functions (relating labels to each other).
  • Latent-Dynamic Conditional Random Fields (LDCRFs): An extension of CRFs that introduces latent (hidden) variables. These hidden variables allow the model to capture "hidden dynamics" within the sequence of observations that are not directly apparent in the observed features. This is achieved by modeling the probability of labels given both the observed features and the latent variables.

The authors also investigate:

  • Feature Engineering: They compare three feature sets:
  • F3: Includes features related to amplitude (ΔA) and duration (ΔTn) within the syllable nucleus, along with a heuristic parameter (m) for pitch movement.
  • F4: Replaces the heuristic 'm' parameter with a novel feature, Γs1,s2, which quantifies pitch movement using a glissando-based metric derived from pitch curve stylization.
  • F5: Combines the features from F4 with total syllable duration (ΔTs) and the ratio of voiced to total duration (V/Ts).
  • Context Extension: The models are evaluated with varying context window sizes (W), ranging from 1 (considering only the current syllable) to 5 (considering the current syllable and its four neighbors).

Datasets

The study utilizes two publicly available speech corpora:

  • SPEECON Corpus (Italian): A subset containing read numbers. This corpus was previously used in related prominence detection studies [3, 5].
  • TIMIT Corpus (English): A subset containing read sentences. This corpus is a standard benchmark for speech research and was used in prior prominence detection work [2].

Both subsets were manually segmented into syllables and annotated for syllabic prominence by an expert linguist using a binary (prominent/non-prominent) notation.

Key Insights and Ideas

  • LDCRFs Outperform CRFs: The most significant finding is that LDCRFs consistently achieve better performance than standard CRFs. This suggests that capturing "hidden dynamics" within the sequence of syllables is crucial for accurate prominence detection, even when using only acoustic features.
  • The Importance of Hidden Dynamics: The authors hypothesize that the advantage of LDCRFs might stem from their ability to model dynamics within sequences of non-prominent syllables. This is a novel perspective, as binary annotation often results in long stretches of non-prominent syllables punctuated by prominent ones.
  • Syllable Length Matters: Syllable length (ΔTn) is identified as a particularly important feature, consistently contributing to better performance when included in the feature sets.
  • Novel Pitch Feature is Promising: The new pitch movement feature (Γs1,s2), which quantifies glissandos, proves to be a valuable replacement for heuristic pitch features (m). It offers a more theoretically grounded approach to capturing pitch dynamics and either improves or maintains performance compared to feature sets using the heuristic parameter.
  • Context Window Size: While context is important, the optimal window size varies between corpora. For SPEECON, a context of up to three syllables is beneficial, whereas for TIMIT, a two-syllable context appears sufficient for maximum performance.

Key Results

  • Consistent Improvement with LDCRFs: LDCRFs systematically outperformed CRFs across all feature sets and context extensions on both SPEECON and TIMIT corpora. For instance, on the TIMIT corpus, the lowest LDCRF performance was comparable to the best CRF performance.
  • Feature Set Performance: Feature set F4 (incorporating the novel pitch feature) and F5 (adding duration features) generally yielded better results than F3 (with the heuristic pitch parameter). On the SPEECON subset, F5 achieved the best performance.
  • Context Impact: Statistical significance tests showed that context extension significantly impacts performance, particularly on the SPEECON corpus.
  • Comparison to Prior Work: The LDCRF results on the SPEECON subset (achieving up to 87.85% F-measure) surpassed previously reported F-measures of 73.3% and 75.1% for similar tasks. On TIMIT, the best LDCRF performance (16.32% error rate) also improved upon prior results (18.64% error rate).

Limitations and Future Work

  • Focus on Acoustic Features: The study exclusively uses acoustic features. Future work could explore the integration of linguistic features (e.g., lexical information, syntactic structure) to further enhance performance.
  • Binary Annotation: The reliance on binary prominence annotation simplifies evaluation but might not fully capture the nuanced nature of prominence. Investigating multi-class prominence levels could be a future direction.
  • Context Dependency: The optimal context window size varied between corpora, suggesting that context requirements might be language- or data-dependent. Further research could explore adaptive context selection.
  • Feature Set Synergy: While F5 showed good results, the authors note that the combination of Γs1,s2 and V/Ts features contains at least the same information as the 'm' parameter, suggesting potential for further feature optimization and understanding of feature interactions.

Summary updated

BibTeX
@inproceedings{cutugno2012investigating,
  author = {Francesco Cutugno and Enrico Leone and Bogdan Ludusan and Antonio Origlia},
  title = {{Investigating syllabic prominence with conditional random fields and latent-dynamic conditional random fields}},
  booktitle = {Interspeech},
  year = {2012},
}
Figure