Multivariate analysis of vocal fatigue in continuous reading

Marie-José Caraty, Claude MontaciéInterspeech 2010

TL;DR

This paper investigates how vocal fatigue affects speech characteristics during prolonged reading. They found that nasal phonemes are surprisingly good indicators of vocal fatigue, outperforming general spectral and prosodic features. This suggests a more nuanced approach is needed for detecting vocal fatigue than previously thought.

What Problem Does This Paper Address?

Vocal fatigue, a common issue for professionals who rely heavily on their voice, is difficult to objectively measure. While previous research has explored vocal fatigue induced by tasks like sleep deprivation or sustained speaking, there's a lack of consensus on which speech features are most indicative of fatigue. This paper aims to address this by systematically analyzing changes in speech during a prolonged, cognitively demanding task: continuous reading for three hours. The authors want to identify specific acoustic and prosodic features that reliably signal vocal fatigue, moving beyond general observations to pinpoint more discriminative markers.

Methods

The researchers designed an experiment where six readers engaged in continuous reading for three hours. They collected speech data and analyzed it in two segments: the beginning (assumed to be non-fatigued) and the end (assumed to be fatigued).

  1. Speech Material: Recordings from a French TV program ("Voyage au bout de la nuit") were used, consisting of continuous reading of literary works.
  2. Error Analysis: To confirm fatigue, they analyzed reading errors (e.g., word substitutions, omissions) and disfluencies (e.g., repetitions, fragments). An increase in these errors from the beginning to the end of the reading session served as an indicator of cognitive and vocal fatigue.
  3. Acoustic Feature Extraction: For each phoneme, they extracted 12 Mel Frequency Cepstral Coefficients (MFCCs). For voiced phonemes, they also extracted fundamental frequency (f0), duration, jitter, shimmer, and the first three formants (f1, f2, f3).
  4. Multivariate Analysis: They employed Multivariate Analysis of Variance (MANOVA) and specifically the Wilks' lambda test to identify which phonemes and features showed significant differences between the non-fatigued and fatigued speech segments. This allowed them to assess the discriminative power of individual phonemes and feature sets.
  5. Parameter Selection: A systematic algorithm was used to identify the least discriminating parameters, progressively removing them to find the most informative subset for detecting vocal fatigue.

Datasets

The study utilized speech corpora from recordings of the Direct 8 TV program "Voyage au bout de la nuit." This program involved continuous reading of French literary works by young actors. The authors created a database of 54,520 word occurrences and 169,042 phoneme occurrences from six readers, each reading for approximately 30 minutes per segment, totaling about 3 hours of speech per reader.

Key Insights and Ideas

  • Nasals as Fatigue Indicators: The most surprising finding is that nasal phonemes (both vowels and consonants, like /m/, /n/, /ŋ/) are significantly more discriminative of vocal fatigue than other phoneme classes. This was consistent across all readers.
  • Effortful Production of Nasals: The authors hypothesize that the increased effort required to control the velum (for nasalization) might be disrupted by prolonged vocal load, leading to more noticeable acoustic changes in these sounds.
  • Velum Dehydration Hypothesis: Similar to vocal folds, the velum might also be affected by dehydration from airflow during prolonged speaking, potentially increasing viscosity and impacting its control.
  • Spectral vs. Prosodic Features: While spectral features (MFCCs) were generally more discriminative than prosodic features (duration, jitter, shimmer), the fundamental frequency (f0) did show some discriminative power.
  • Phoneme-Specific Changes: Vocal fatigue doesn't affect all speech sounds equally. Changes are phoneme-dependent, suggesting that a "phoneme spotting system" might be more effective for fatigue detection than a global analysis.
  • Global vs. Phoneme-Class Analysis: Analyzing features at the phoneme class level provided more insight than looking at overall speech characteristics.

Key Results

  • Nasals Dominate Discrimination: Nasal phonemes consistently ranked highest in discriminative power for vocal fatigue, as indicated by high Wilks' lambda values.
  • MFCCs Outperform Prosody: When considering only spectral features (MFCCs), the analysis identified 22 phonemes as significantly discriminative.
  • Combined Features Improve Discrimination: Including prosodic and spectral features (19 parameters total) further improved discrimination, with 18 phonemes showing significant Wilks' lambda values.
  • Parameter Reduction: The parameter selection process revealed that the first cepstral coefficients (c1, c2, c8, c3, c10, c5, c4, c6) were the most discriminative, while formants (f1, f2, f3) and some prosodic features (d, sh, jt) were less so.
  • Error Rate Increase: Reading errors and disfluencies increased significantly from the initial reading segment (s1) to the final segment (s6), confirming the induction of cognitive and vocal fatigue. For one reader (R3), the error rate tripled.

Limitations and Future Work

  • Reader Independence: While six readers were used, the authors acknowledge that the results might still be influenced by individual speaking styles. Future work should include more speakers and potentially other languages to test the robustness of the findings, especially regarding nasal phonemes.
  • Fatigue Induction Method: The study used prolonged reading. It would be beneficial to investigate if fatigue induced by other means, such as sleep deprivation, yields similar results.
  • Real-World Application: The authors suggest exploring the feasibility of a vocal fatigue detector based on these findings, correlating acoustic results with reading errors and disfluencies.
  • Specific Phoneme Exclusions: The parameter selection process excluded certain phonemes due to low occurrence or inherent variability, meaning conclusions about these specific phonemes cannot be drawn from that analysis.

Summary updated

BibTeX
@inproceedings{caraty2010multivariate,
  author = {Marie-José Caraty and Claude Montacié},
  title = {{Multivariate analysis of vocal fatigue in continuous reading}},
  booktitle = {Interspeech},
  year = {2010},
}
Figure