TL;DR
This paper introduces UltrasonicGS, a novel system for contactless gesture and sign language recognition using ultrasonic signals. It addresses the challenge of limited data and variable sequence lengths by employing GANs for data augmentation and CTC for sequence alignment, achieving impressive recognition rates for both single and continuous gestures, including Chinese sign language.
What Problem Does This Paper Address?
The COVID-19 pandemic highlighted the need for contactless interaction methods. While gesture recognition offers a solution, existing methods face limitations. Vision-based systems are sensitive to lighting and privacy concerns, while sensor-based systems require wearable devices. Wi-Fi-based methods can be affected by interference. Furthermore, recognizing the sign language of deaf individuals presents a specific challenge due to the complexity of movements and the scarcity of acoustic-based recognition systems for languages like Chinese Sign Language. Existing approaches also struggle with handling continuous gestures and sign language sequences of varying lengths.
Methods
UltrasonicGS employs a multi-stage approach:
- Data Collection: Ultrasonic signals are emitted by speakers and received by a microphone, capturing echo signals reflecting gesture movements.
- Data Pre-processing: Raw echo signals are filtered (Butterworth bandpass) and processed using Short-Time Fourier Transform (STFT) to extract Doppler shifts, followed by Gaussian smoothing.
- Data Augmentation: Generative Adversarial Networks (GANs) are used to artificially expand the dataset, creating more diverse samples and improving model generalization without human intervention.
- Feature Extraction: A ResNet34 network is utilized to extract multi-scale semantic features from the processed spectrograms, generating feature vectors.
- Gesture Classification:
- For single gestures, a Bi-directional Long Short-Term Memory (Bi-LSTM) network is used for classification.
- For continuous gestures and sign language, the Connectionist Temporal Classification (CTC) algorithm is added after the Bi-LSTM network. CTC is crucial for handling the variable lengths and alignment issues inherent in sequential data.
Datasets
The paper describes data collection in both laboratory and corridor environments.
- Single Gestures: 15 single gestures were collected from 12 volunteers (6 male, 6 female).
- Continuous Gestures & Sign Language: 6 continuous gestures and 6 sign language gestures were collected from 4 volunteers (2 male, 2 female).
- The dataset includes data under various influencing factors like distance, speed, noise, and angle.
- A new open-source dataset has been created and made available.
Key Insights and Ideas
- Acoustic Sensing for Fine-Grained Gestures: The paper demonstrates that ultrasonic signals can capture fine-grained gesture movements, extending beyond simple whole-hand gestures.
- GANs for Data Scarcity: Using GANs for data augmentation is shown to be effective in improving model performance and generalization, especially when dealing with limited real-world data.
- CTC for Sequence Alignment: The integration of CTC is a critical innovation for handling the inherent challenges of variable-length input and output sequences in continuous gesture and sign language recognition.
- Bridging the Gap in Chinese Sign Language: The work specifically addresses the lack of acoustic-based recognition for Chinese Sign Language, a significant contribution.
- Robustness to Environmental Factors: The system shows good performance across different environments (lab vs. corridor) and is relatively robust to noise and some forms of interference.
Key Results
- Single Gesture Recognition: Achieved a combined recognition rate of 98.8% for 15 single gestures.
- Continuous Gesture Recognition: Achieved an average correct recognition rate of 92.4% for six sets of continuous gestures.
- Sign Language Gesture Recognition: Achieved an average correct recognition rate of 86.3% for six sets of sign language gestures (specifically Chinese Sign Language).
- The system maintains good performance up to a distance of 0.5 m.
- The method demonstrates robustness to low-frequency and ultrasonic noise.
Limitations and Future Work
- Sign Language Recognition Accuracy: While the system performs well, the authors acknowledge that improving recognition accuracy for sign language datasets is an area for future work.
- Device Replacement: The current system uses specialized hardware (ASDP). Future work aims to replace this with a standard cell phone to make the technology more accessible.
- Speech/Text Conversion: The authors plan to develop functions for sign language speech and text conversion to enhance human-computer interaction.
- Chinese Sign Language Complexity: The paper notes that Chinese Sign Language recognition is more complex due to homophones and split words, which may contribute to the slightly lower accuracy compared to English sign language recognition in some benchmarks.
Summary updated