Abstract
Stuttering is a complex neurological condition characterized by speech disfluency, yet its physiological precursors begin well before any audible sound is produced. This paper presents a self-supervised deep learning framework designed to predict disfluency by analyzing facial micro-expressions during the "pre-speech" phase. By training on unlabeled facial muscle dynamics through specialized pretext tasks, the model achieves superior accuracy and robustness compared to traditional supervised methods. Furthermore, the integration of explainable AI (XAI) provides clinical insights into the involuntary muscle "tells" that precede a stutter, offering a path toward personalized diagnostic and biofeedback tools.
Introduction
Speech disfluency is often studied through its audible manifestations, but the cognitive and motor preparation for speech occurs in the seconds leading up to vocalization. This preparation phase involves subtle, involuntary facial muscle movements known as micro-expressions. Capturing these patterns presents significant challenges: medical data is scarce and expensive to label, and clinical applications require "white-box" interpretability to understand the underlying pathology.
This research addresses these challenges by leveraging self-supervised learning to extract meaningful features from limited datasets and utilizing DeepSHAP to map model predictions back to human anatomy.
Methodology
The proposed framework utilizes a two-stage pipeline: self-supervised pre-training followed by downstream classification with explainability.
Self-Supervised Pre-training
To overcome data scarcity, the model first learns the temporal dynamics of facial movements using two pretext tasks. First, the Micro-expression Window Finder requires the model to identify a specific 100ms window within a sequence of facial Action Units (AUs). Second, Signal Transformation Identification tasks the model with identifying artificial corruptions (e.g., Gaussian noise) applied to the data, enhancing its robustness to real-world video artifacts like head jitters.
Downstream Classification and Explainability
The pre-trained model is fine-tuned on a labeled dataset of "Fluent" vs. "Disfluent" trials. To ensure clinical utility, the researchers employed DeepSHAP to generate "Attribution Maps," which highlight the specific facial muscles and timeframes that contribute most to the model's classification.
Experimental Setup
The study utilized the AWS (Adults-Who-Stutter) Dataset, a multimodal collection of 3,700 trials. Each trial consists of 1500ms of video captured at 58 FPS immediately preceding speech. Using the OpenFace toolkit, 17 Action Units (8 upper face, 9 lower face) were extracted to serve as the model's input.
Results and Key Insights
The self-supervised approach (CNN-A) demonstrated significant advantages over traditional supervised learning:
- Performance Gains: The model achieved a minimum 2.51% accuracy improvement over fully-supervised baselines.
- Data Efficiency: When trained on only 10% of the labeled data, the self-supervised model maintained an AUC of ~0.77, whereas supervised models failed to generalize.
- Physiological "Tells": The XAI component revealed that the first 500ms of speech preparation are critical. Significant activity in the upper lip raiser (AU 10), lip stretcher (AU 20), and lip corner puller (AU 12) was identified as a precursor to disfluency. Additionally, individuals with higher stuttering rates exhibited increased blinking (AU 45) during disfluent preparation phases.
Conclusion
This work demonstrates that self-supervised learning can effectively decode the subtle facial dynamics preceding speech disfluency. By providing interpretable maps of muscle activity, the framework moves beyond simple prediction toward a deeper understanding of the neurological "motor overflow" associated with stuttering. Future research will focus on personalizing these models to individual facial signatures and developing real-time biofeedback systems for clinical intervention.
Summary updated