Automatic Recognition of Gesture Identity and Onset of Cues-Speech

Annahita Sarré, Hagar Salpeter, Deliane Bechar, Laurent Cohen, Yair Lakretz

Abstract

Cued speech is a communication system based on hand gestures used in certain communities of deaf people around the world. Hand gestures in cued speech convey complementary information to that available from lip-reading alone, which helps to communicate accurate phonological information. Cued speech provides several unique opportunities for scientific research, such as studying phonological processing via the visual modality. However, cued speech has been studied only scarcely since its invention, and there are only a few empirical datasets and standardized methods available to the scientific community. Here, we suggest several contributions to advance research in the field: (1) A new dataset on cued speech annotated for various linguistic features, (2) a new approach to automatically identify gesture identity and gesture onset from raw videos, using cued-speech-specific features, which achieves relatively high performance (AUC-identity > 0.95, Error-onset = 93ms), and (3) additional insights into the relationship between sound and gesture production in cued speech, showing that syllable acoustic onset precedes gesture onset by around 150ms on average and that this time difference is more sensitive to consonant rather than to vowel identity. We make the new dataset and all associated tools publicly available.

Introduction

Cued speech (CS) is a visual communication system designed to enhance lip-reading for the deaf and hard-of-hearing. By combining hand gestures—specifically hand shapes and positions—with natural lip movements, it resolves phonological ambiguities (e.g., distinguishing /p/, /b/, and /m/). Despite its utility, research in the field has been hindered by a lack of standardized datasets and the technical complexity of automated recognition systems.

Overview of the French Cued Speech system, illustrating how hand shapes represent consonants and positions relative to the face represent vowels.

Methods

This study introduces a new French cued speech dataset and a streamlined Automatic Cued Speech Recognition (ACSR) pipeline. The pipeline utilizes MediaPipe for markerless 3D landmark extraction from raw 4K video, avoiding the need for physical markers on subjects.

Instead of relying solely on complex deep learning architectures, the authors employed domain-specific feature engineering. They calculated features representing the mechanics of cued speech, such as finger "openness" and hand-to-face distance. These features were then used to train linear Support Vector Machines (SVM) and Logistic Regression models. To pinpoint gesture timing, a novel "Change-Rate" measure was developed, combining hand velocity with decoder confidence levels to detect gesture onsets.

Results

The recognition system achieved high performance, with an AUC > 0.95 for gesture identity. Specifically, the SVM reached 97.7% accuracy for hand position and 94.5% for hand shape. Gesture onset detection showed an average error of only 93ms, significantly lower than the average duration between gestures.

A key finding of the study is the temporal relationship between sound and gesture. Analysis revealed a consistent "lag": the acoustic signal typically precedes the visual gesture by approximately 150ms (mean = 149ms, SD = 186ms).

Temporal analysis results: (A) acoustic-phonetic alignment, (B) onset detection metrics, (C) distribution of the acoustic-visual lag, and (D) lag variation across vowels and consonants.

Discussion and Conclusion

The sound-gesture lag was found to be more sensitive to consonant identity than vowel identity. This suggests that the motor planning for hand shapes (consonants) is more tightly coupled to speech production than hand positions (vowels). While the current dataset is limited to two experienced cuers, the robustness of the results and the availability of the open-source tools provide a significant step forward for the study of multimodal phonological processing. Future work will focus on expanding the dataset to include more diverse users and natural conversational environments.

Summary updated

Abstract

Cued speech is a communication system based on hand gestures used in certain communities of deaf people around the world. Hand gestures in cued speech convey complementary information to that available from lip-reading alone, which helps to communicate accurate phonological information. Cued speech provides several unique opportunities for scientific research, such as studying phonological processing via the visual modality. However, cued speech has been studied only scarcely since its invention, and there are only a few empirical datasets and standardized methods available to the scientific community. Here, we suggest several contributions to advance research in the field: (1) A new dataset on cued speech annotated for various linguistic features, (2) a new approach to automatically identify gesture identity and gesture onset from raw videos, using cued-speech-specific features, which achieves relatively high performance (AUC-identity>0.95), $Error-onset=93ms$), and (3) additional insights into the relationship between sound and gesture production in cued speech, showing that syllable acoustic onset precedes gesture onset by around 150ms on average and that this time difference is more sensitive to consonant rather than to vowel identity. We make the new dataset and all associated tools publicly available.
BibTeX
@inproceedings{sarre2024automatic,
  author = {Annahita Sarré and Hagar Salpeter and Deliane Bechar and Laurent Cohen and Yair Lakretz},
  title = {{Automatic Recognition of Gesture Identity and Onset of Cues-Speech}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2024},
  doi = {10.1109/icassp48485.2024.10446932},
}
Figure