Analysis and detection of speech under sleep deprivation

Tin Lay Nwe, Haizhou Li, Minghui Dong

Abstract

Sleep deprivation (SD) significantly impacts vocal characteristics, leading to a "flattened" or monotonous voice that can degrade the performance of automated speech recognition systems. This paper presents a novel feature extraction method, Two-Layer Cascaded-Subband Filters (TLCSF), designed to capture subtle variations in pitch and harmonic patterns. By integrating these acoustic features with behavioral response-time analysis, the proposed system achieves an 86.5% accuracy in distinguishing normal speech from sleep-deprived speech, outperforming standard Mel-Frequency Cepstral Coefficients (MFCCs).

Introduction

In high-stakes environments such as military aviation, emergency response, and long-haul transportation, sleep deprivation is a critical safety risk. Fatigue alters the physiological production of speech, making it slurred, mumbled, or notably "flat." These changes are not merely a nuisance; they can cause critical communication failures when automated systems fail to recognize exhausted voices.

Standard acoustic features like MFCCs are optimized for general speech recognition and often overlook the specific markers of exhaustion. This research identifies the acoustic and behavioral shifts that occur after 60 hours of wakefulness and introduces a classifier capable of reliably detecting these states.

Methodology

The core technical contribution is the development of Pitch and Harmonic frequency Spectral Coefficients (PHSC), which are derived through a specialized two-step process:

Two-Layer Cascaded-Subband Filters (TLCSF)

Unlike the standard triangular filters used in MFCCs, the TLCSF employs a hierarchical filter bank. The first layer consists of seven overlapping filters covering the pitch and harmonic ranges (60Hz to 16kHz). The second layer further subdivides these into four non-overlapping rectangular filters. This architecture allows the system to track local energy maxima with high precision, making it sensitive to the subtle shifts in pitch and harmonics that characterize healthy speech.

The Two-Layer Cascaded-Subband Filter (TLCSF) architecture showing the hierarchical division of frequency bands.

Behavioral and Statistical Modeling

Beyond acoustics, the researchers incorporated Response-Time (RT)—the latency between an instruction and a response. Using the "Elias Coefficient," the team statistically modeled how these response distributions shift as fatigue sets in. These combined features were then processed by a Hidden Markov Model (HMM) with four states and two Gaussian mixtures to classify speech as either "Normal" (NS) or "Sleep Deprived" (SDS).

Experimental Setup

The study utilized the DCIEM Map Task Corpus, featuring 36 Canadian adults (34 male, 2 female) engaged in unscripted, naturalistic dialogue. The experiment focused on a "placebo" group (no stimulants) over a 60-hour sleepless period, resulting in a dataset of 216 dialogues.

Results and Discussion

The "Flattened Voice" Phenomenon

A key finding is that sleep-deprived speech exhibits significantly less variation in pitch and harmonic patterns over time. In a healthy state, human prosody modulates naturally; however, as vocal muscles lose tone due to exhaustion, the "melody" of speech stabilizes into a monotonous drone.

Comparison of spectral frames: (a) shows the natural shifting of peaks in normal speech, while (b) illustrates the stable, 'flattened' harmonic structure typical of sleep deprivation.

Classification Performance

The proposed PHSC features significantly outperformed traditional methods, particularly in identifying sleep-deprived states where standard techniques often struggle.

Feature Type Normal Speech Accuracy Sleep Deprived Accuracy Average Accuracy
PHSC (Proposed) 93.4% 79.6% 86.5%
LPCC 93.5% 77.4% 85.4%
MFCC 93.5% 71.2% 82.3%

The 8.4% improvement over MFCCs in detecting sleep-deprived speech (79.6% vs 71.2%) suggests that the cascaded subband approach is uniquely suited to capturing the spectral "thinning" caused by fatigue.

Conclusion and Future Work

While the TLCSF method provides a robust framework for fatigue detection, several areas for improvement remain. The current dataset is heavily male-skewed, potentially limiting generalizability for female voices. Furthermore, the study treats sleep deprivation as a binary state rather than a gradient. Future research should focus on detecting the progression of fatigue and testing the system's resilience against real-world environmental noise, such as that found in cockpits or heavy machinery cabins.

Summary updated

BibTeX
@inproceedings{nwe2006analysis,
  author = {Tin Lay Nwe and Haizhou Li and Minghui Dong},
  title = {{Analysis and detection of speech under sleep deprivation}},
  booktitle = {Interspeech},
  year = {2006},
  doi = {10.21437/interspeech.2006-509},
}
Figure