Voice Quality and Between-Frame Entropy for Sleepiness Estimation

Vijay Ravi, Soo Jin Park, Amber Afshan, Abeer AlwanInterspeech 2019

TL;DR

This paper introduces a novel approach to automatically estimate a person's degree of sleepiness using voice analysis. By combining traditional acoustic features with a new "between-frame entropy" measure that captures speaking rate, and employing robust data preprocessing and fusion techniques, the system achieves significant improvements over existing methods. This work has implications for safety-critical applications like driver-assistance systems.

What Problem Does This Paper Address?

Monitoring a person's alertness, particularly their level of sleepiness, is crucial for safety in various high-stakes scenarios, such as driving or operating complex machinery. Sleepiness significantly impacts cognitive and physiological functions, which in turn manifest in subtle changes in speech production. These changes can include a slower speaking rate, altered fundamental frequency, and modified vocal tract acoustics. While previous research has explored detecting sleepiness from speech, this paper focuses on estimating the degree of sleepiness, rather than just classifying it as "sleepy" or "not sleepy." The challenge lies in capturing these subtle, often idiosyncratic, vocal changes reliably and building a system that is robust to variations in speech style and individual speaking patterns.

Methods

The authors propose a system that leverages three main components:

  • Voice Quality Features (VQual): Inspired by psycho-acoustic models, these features capture aspects of the voice source and vocal tract, such as fundamental frequency (Fo), formant frequencies and amplitudes, and spectral tilt. These are hypothesized to be sensitive to the physiological changes associated with sleepiness.
  • Between-Frame Entropy: This novel feature is introduced as an instantaneous measure of speaking rate. It quantifies how rapidly the spectral characteristics of the speech signal change from one frame to the next. A higher entropy suggests more rapid changes, indicative of a faster speaking rate. This is important because sleepiness is known to affect speaking speed.
  • i-vector Representation: Instead of using simple statistical summaries of acoustic features, the authors employ the i-vector framework. This technique represents an entire utterance as a low-dimensional vector that captures the underlying distribution of acoustic features, providing a more robust and discriminative representation.
  • Outlier Elimination: Recognizing that sleepiness effects can be highly individual and sometimes manifest in unexpected ways, the authors implement an outlier detection and removal step on the training data. This process uses between-frame entropy-based i-vectors to identify and discard samples that deviate significantly, thereby improving the robustness of the sleepiness predictor.
  • Support Vector Regression (SVR): A regression model is trained to predict the continuous degree of sleepiness.
  • Score-Level Fusion: Individual predictors trained on each feature set (VQual, MFCCs, and ComParE16) are combined at the score level to leverage their complementary information.

Datasets

  • SLEEP Corpus: A subset of the Duesseldorf Sleepy Language corpus was used for evaluating the sleepiness estimation system. This corpus contains speech recordings from German speakers, with self-assessed sleepiness ratings on the Karolinska Sleepiness Scale (KSS), ranging from 1 (extremely alert) to 9 (very sleepy).
  • NIST SRE 04, 05, 06, 08 and Switchboard II corpus phase 2: These large databases, totaling over 3,000 hours of speech from thousands of speakers, were used to train the Universal Background Model (UBM) and i-vector extractor.

Key Insights and Ideas

  • Between-frame entropy as a proxy for speaking rate: The paper provides strong empirical evidence (Pearson's correlation of 0.71) that between-frame entropy is a reliable indicator of instantaneous speech rate, a key physiological correlate of sleepiness.
  • Robustness through outlier elimination: Removing outlier speech samples from the training data significantly improves the performance of sleepiness prediction models across different feature sets and utterance representations. This highlights the idiosyncratic nature of sleepiness effects on speech.
  • i-vectors offer superior utterance representation: The i-vector framework consistently outperforms simpler statistics vectors for representing utterances, leading to substantial performance gains for all acoustic features. This suggests that capturing the distribution of acoustic features is more effective than just summarizing them.
  • Complementarity of feature sets: Fusing different feature sets, particularly voice quality features with others like MFCCs and ComParE16, leads to improved sleepiness estimation. This indicates that these features capture different, yet valuable, aspects of sleepiness-related vocal changes.

Key Results

  • The proposed system, which fuses VQual, MFCCs, and ComParE16 features using i-vector representation and outlier elimination, achieved a relative improvement of 19.5% on the development dataset and 5.4% on the test dataset compared to the baseline ComParE16 system.
  • Outlier elimination consistently improved the Spearman's correlation coefficient (ρ) for individual feature sets. For example, for MFCCs with i-vectors, ρ increased from 0.248 to 0.252.
  • Score fusion of VQual with ComParE16 improved the ρ from 0.251 to 0.283 on the development set.
  • The best performing individual system on the development set using i-vectors was the MFCCs-based system with an outlier elimination (ρ = 0.252).

Limitations and Future Work

  • Speech Task Metadata: The study acknowledges that the varying effects of sleepiness on speech rate (e.g., negative correlation for read speech vs. positive for spontaneous speech) could not be fully investigated due to the lack of speech task metadata in the dataset.
  • UBM and i-vector Extractor Training: The authors suggest that training the UBM and i-vector extractor on a German speech database could lead to more reliable utterance representations, as the current study used a multi-lingual database.
  • Feature Selection and Dimensionality Reduction: Exploring feature selection or dimensionality reduction techniques could further analyze the contribution of individual features and potentially improve system performance.
  • Adaptive Strategies: Developing adaptive strategies to compensate for speaking style and speaker variability is proposed as a promising direction, given that sleepiness influence can vary based on these factors.

Summary updated

BibTeX
@inproceedings{ravi2019voice,
  author = {Vijay Ravi and Soo Jin Park and Amber Afshan and Abeer Alwan},
  title = {{Voice Quality and Between-Frame Entropy for Sleepiness Estimation}},
  booktitle = {Interspeech},
  year = {2019},
}
Figure